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Chapter  1:  Getting  started  with  algorithms 

Section  1.1:  A  sample  algorithmic  problem 

An  algorithmic  problem  is  specified  by  describing  the  complete  set  of  instances  it  must  work  on  and  of  its  output 
after  running  on  one  of  these  instances.  This  distinction,  between  a  problem  and  an  instance  of  a  problem,  is 
fundamental.  The  algorithmic  problem  known  as  sorting  is  defined  as  follows:  [Skiena:2008:ADM:1 41 021 9] 

•  Problem:  Sorting 

•  Input:  A  sequence  of  n  keys,  a_1 ,  a_2,  ...,  a_n. 

•  Output:  The  reordering  of  the  input  sequence  such  that  a '  _l  <=  a'_2  <=  ...  <=  a '  _  {n  - 1 }  <=  a'_n 

An  instance  of  sorting  might  be  an  array  of  strings,  such  as  {  Haskell,  Emacs  }  or  a  sequence  of  numbers  such  as 

{  154,  245,  1337  }. 

Section  1.2:  Getting  Started  with  Simple  Fizz  Buzz  Algorithm  in 
Swift 

For  those  of  you  that  are  new  to  programming  in  Swift  and  those  of  you  coming  from  different  programming  bases, 
such  as  Python  or  Java,  this  article  should  be  quite  helpful.  In  this  post,  we  will  discuss  a  simple  solution  for 
implementing  swift  algorithms. 

Fizz  Buzz 

You  may  have  seen  Fizz  Buzz  written  as  Fizz  Buzz,  FizzBuzz,  or  Fizz-Buzz;  they're  all  referring  to  the  same  thing.  That 
"thing"  is  the  main  topic  of  discussion  today.  First,  what  is  FizzBuzz? 

This  is  a  common  question  that  comes  up  in  job  interviews. 

Imagine  a  series  of  a  number  from  1  to  1 0. 

123456789  10 

Fizz  and  Buzz  refer  to  any  number  that's  a  multiple  of  3  and  5  respectively.  In  other  words,  if  a  number  is  divisible 
by  3,  it  is  substituted  with  fizz;  if  a  number  is  divisible  by  5,  it  is  substituted  with  buzz.  If  a  number  is  simultaneously 
a  multiple  of  3  AND  5,  the  number  is  replaced  with  "fizz  buzz."  In  essence,  it  emulates  the  famous  children  game 
"fizz  buzz". 

To  work  on  this  problem,  open  up  Xcode  to  create  a  new  playground  and  initialize  an  array  like  below: 

//  for  example 

let  number  =  [1,2, 3, 4, 5] 

//  here  3  is  fizz  and  5  is  buzz 

To  find  all  the  fizz  and  buzz,  we  must  iterate  through  the  array  and  check  which  numbers  are  fizz  and  which  are 
buzz.  To  do  this,  create  a  for  loop  to  iterate  through  the  array  we  have  initialised: 

for  num  in  number  { 

//  Body  and  calculation  goes  here 

} 

After  this,  we  can  simply  use  the  "if  else"  condition  and  module  operator  in  swift  ie  -  %  to  locate  the  fizz  and  buzz 
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for  num  in  number  { 
if  num  %  3  ==  0  { 

print("\(num)  fizz") 

}  else  { 

print (num) 

} 

} 

Great!  You  can  go  to  the  debug  console  in  Xcode  playground  to  see  the  output.  You  will  find  that  the  "fizzes"  have 
been  sorted  out  in  your  array. 

For  the  Buzz  part,  we  will  use  the  same  technique.  Let's  give  it  a  try  before  scrolling  through  the  article  —  you  can 
check  your  results  against  this  article  once  you've  finished  doing  this. 

for  num  in  number  { 
if  num  %  3  ==  0  { 
print("\(num)  fizz") 

}  else  if  num  %  5  ==  0  { 
print("\(num)  buzz") 

}  else  { 
print (num) 

} 

} 

Check  the  output! 

It's  rather  straight  forward  —  you  divided  the  number  by  3,  fizz  and  divided  the  number  by  5,  buzz.  Now,  increase 
the  numbers  in  the  array 

let  number  =  [1 ,2,3,4,5,6,7,8,9,10, 11 ,12, 13,14, 15] 

We  increased  the  range  of  numbers  from  1  -1 0  to  1  -1 5  in  order  to  demonstrate  the  concept  of  a  "fizz  buzz."  Since  1 5 
is  a  multiple  of  both  3  and  5,  the  number  should  be  replaced  with  "fizz  buzz."  Try  for  yourself  and  check  the  answer! 

Here  is  the  solution: 

for  num  in  number  { 

if  num  %  3  ==  0  &&  num  %  5  ==  0  { 

print("\(num)  fizz  buzz") 

}  else  if  num  %  3  ==  0  { 
print("\(num)  fizz") 

}  else  if  num  %  5  ==  0  { 
print( " \(num)  buzz") 

}  else  { 
print (num) 

} 

} 

Wait.. .it's  not  over  though!  The  whole  purpose  of  the  algorithm  is  to  customize  the  runtime  correctly.  Imagine  if  the 
range  increases  from  1  -1 5  to  1  -1 00.  The  compiler  will  check  each  number  to  determine  whether  it  is  divisible  by  3 
or  5.  It  would  then  run  through  the  numbers  again  to  check  if  the  numbers  are  divisible  by  3  and  5.  The  code  would 
essentially  have  to  run  through  each  number  in  the  array  twice  —  it  would  have  to  runs  the  numbers  by  3  first  and 
then  run  it  by  5.  To  speed  up  the  process,  we  can  simply  tell  our  code  to  divide  the  numbers  by  1 5  directly. 

Here  is  the  final  code: 

for  num  in  number  { 
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if  num  %  15  ==  0  { 

print("\(num)  fizz  buzz") 
}  else  if  num  %  3  ==  0  { 
print("\(num)  fizz") 

}  else  if  num  %  5  ==  0  { 
print("\(num)  buzz") 

}  else  { 
print (num) 

} 


As  Simple  as  that,  you  can  use  any  language  of  your  choice  and  get  started 
Enjoy  Coding 


GoalKicker.com  -  Algorithms  Notes  for  Professionals 


4 


Chapter  2:  Algorithm  Complexity 

Section  2.1:  Big-Theta  notation 

Unlike  Big-0  notation,  which  represents  only  upper  bound  of  the  running  time  for  some  algorithm,  Big-Theta  is  a 
tight  bound;  both  upper  and  lower  bound.  Tight  bound  is  more  precise,  but  also  more  difficult  to  compute. 

The  Big-Theta  notation  is  symmetric:  f(x)  =  0(g(x))  <=>  g(x)  =  0(f(x)) 

An  intuitive  way  to  grasp  it  is  that  f  (x)  =  e(g(x) )  means  that  the  graphs  of  f(x)  and  g(x)  grow  in  the  same  rate,  or 
that  the  graphs  'behave'  similarly  for  big  enough  values  of  x. 

The  full  mathematical  expression  of  the  Big-Theta  notation  is  as  follows: 

0(f(x))  =  (g:  NO  ->  R  and  cl ,  c2,  nO  >  0,  where  cl  <  abs(g(n)  /  f(n)),  for  every  n  >  nO  and  abs  is  the  absolute  value  } 

An  example 

If  the  algorithm  for  the  input  n  takes  42nA2  +  25n  +  4  operations  to  finish,  we  say  that  is  0(nA2),  but  is  also  0(nA3) 
and  O(nA100).  However,  it  is  0(nA2)  and  it  is  not  0(nA3),  0(nA4)  etc.  Algorithm  that  is  0(f  (n) )  is  also  0(f  (n) ),  but 
not  vice  versa! 

Formal  mathematical  definition 

0(g  (x) )  is  a  set  of  functions. 

0(g(x))  =  {f(x)  such  that  there  exist  positive  constants  cl,  c2,  N  such  that  0  <=  c1*g(x)  <=  f(x) 

<=  c2*g(x)  for  all  x  >  N} 

Because  0(g(x) )  is  a  set,  we  could  write  f(x)  e  0(g(x))to  indicate  that  f  (x)  is  a  member  of  0(g(x) ).  Instead,  we 
will  usually  write  f  (x)  =  0(g(x) )  to  express  the  same  notion  -  that's  the  common  way. 

Whenever  0(g(x) )  appears  in  a  formula,  we  interpret  it  as  standing  for  some  anonymous  function  that  we  do  not 
care  to  name.  For  example  the  equation  T(n)  =  T(n/2)  +  0(n),  means  T(n)  =  T(n/2)  +  f  (n)  where  f  (n)  is  a 
function  in  the  set  0(n). 

Let  f  and  g  be  two  functions  defined  on  some  subset  of  the  real  numbers.  We  write  f(x)  =  0(g(x))  as 
x->inf  inity  if  and  only  if  there  are  positive  constants  K  and  L  and  a  real  number  x0  such  that  holds: 

K  |  g ( x )  |  <=  f  (x)  <=  L  |  g  (x)  |  for  all  x  >=  x0. 

The  definition  is  equal  to: 

f(x)  =  0(g (x) )  and  f(x)  =  Q(g(x)) 

A  method  that  uses  limits 

if  limit  (x->inf  inity)  f(x)/g(x)  =  c  e  (0,a))  i.e.  the  limit  exists  and  it's  positive,  then  f(x)  =  0(g(x)) 


Common  Complexity  Classes 


Name 

Notation 

n  =  10 

n  =  100 

Constant 

9(1) 

1 

1 

Logarithmic  0(log(n)) 

3 

7 

Linear 

0(n) 

10 

100 

Linearithmic  0(n*log(n)) 

30 

700 
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Quadratic  0(nA2) 
Exponential  0(2An) 
Factorial  0(n!) 


100  10  000 
1  024  1 .267650e+  30 
3  628  800  9.332622e+1  57 


Section  2.2:  Comparison  of  the  asymptotic  notations 

Let  f  (n)  and  g(n)  be  two  functions  defined  on  the  set  of  the  positive  real  numbers,  c,  cl ,  c2,  n0  are  positive  real 
constants. 


Notation 


f(n)  =  0(g(n)) 


Formal 

definition 


]  n0  >  0  :  V  n  >  n0,  0  <  f(n)  <  c  g(n) 


Analogy 
between  the 
asymptotic 
comparison  a  <  b 
off,  g  and 
real  numbers 
a,  b 

Example  7n  +  10  =  0(nA2  +  n  -  9) 


a 


f(n)  =  0(g(n)) 


cg(n) 


Graphic 

interpretation 


f(n)  =  Q(g(n)) 


3  c  >  0,  ]  n0  >  0  :  V  n  >  n0,  0  <  c  g(n)  <  f(n) 


f(n)  =  0(g(n)) 


3  cl ,  c2  >  0,  ]  n0  >  0  :  V  n  >  n0, 
f(n)  <  c2  g(n) 


f(n)  =  f(n)  = 
o(g(n))  co(g(n)) 

V  c  >  V  c  > 
0,  3  0,  3 

n0  >  0  n0  >  0 

0  <  cl  g  (n)  <  :  V  n  :  V  n 
>  n0,  >  n0, 

0  s  0  s  c 

f(n)  <g(n)  < 
c  g(n)  f(n) 


n*3  -  34  =  Q(10nA2  -  7n  +  1 )  1/2  nA2  -  7n  =  0(n«2) 


The  asymptotic  notations  can  be  represented  on  a  Venn  diagram  as  follows: 


Links 

Thomas  H.  Cormen,  Charles  E.  Leiserson,  Ronald  L.  Rivest,  Clifford  Stein.  Introduction  to  Algorithms. 

Section  2.3:  Big-Omega  Notation 

Q-notation  is  used  for  asymptotic  lower  bound. 

Formal  definition 

Let  f  (n)  and  g(n)  be  two  functions  defined  on  the  set  of  the  positive  real  numbers.  We  write  f  (n)  =  Q(g(n) )  if 
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there  are  positive  constants  c  and  n0  such  that: 

0  <  c  g(n)  <  f(n)  for  all  n  >  n0. 

Notes 

f  (n)  =  Q(g(n) )  means  that  f  (n)  grows  asymptotically  no  slower  than  g(n).  Also  we  can  say  about  Q(g(n) )  when 
algorithm  analysis  is  not  enough  for  statement  about  0(g(n))  or/andO(g(n)). 

From  the  definitions  of  notations  follows  the  theorem: 

For  two  any  functions  f(n)  and  g(n)  we  have  f(n)  =  0(g(n))  if  and  only  if  f(n)  =  0(g(n))  and  f(n)  =  Q(g(n)). 
Graphically  Q-notation  may  be  represented  as  follows: 


For  example  lets  we  have  f(n)  =  3nA2  +  5n  4.  Then  f(n)  =  Q(nA2).  It  is  also  correct  f(n)  =  O(n),  or  even  f  (n) 

=  0(1). 

Another  example  to  solve  perfect  matching  algorithm  :  If  the  number  of  vertices  is  odd  then  output  "No  Perfect 
Matching"  otherwise  try  all  possible  matchings. 

We  would  like  to  say  the  algorithm  requires  exponential  time  but  in  fact  you  cannot  prove  a  0(nA2)  lower  bound 
using  the  usual  definition  of  Q  since  the  algorithm  runs  in  linear  time  for  n  odd.  We  should  instead  define 
f  (n)  =Q(g(n) )  by  saying  for  some  constant  c>0,  f  (n)>  c  g(n)  for  infinitely  many  n.  This  gives  a  nice 
correspondence  between  upper  and  lower  bounds:  f(n)=Q(g(n))ifff(n)  !  =  o(g(n) ). 

References 

Formal  definition  and  theorem  are  taken  from  the  book  "Thomas  FI.  Cormen,  Charles  E.  Leiserson,  Ronald  L.  Rivest, 
Clifford  Stein.  Introduction  to  Algorithms". 
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Chapter  3:  Graph 

A  graph  is  a  collection  of  points  and  lines  connecting  some  (possibly  empty)  subset  of  them.  The  points  of  a  graph 
are  called  graph  vertices,  "nodes"  or  simply  "points."  Similarly,  the  lines  connecting  the  vertices  of  a  graph  are  called 
graph  edges,  "arcs"  or  "lines." 

A  graph  G  can  be  defined  as  a  pair  (V,E),  where  V  is  a  set  of  vertices,  and  E  is  a  set  of  edges  between  the  vertices  E  £ 
{(u,v)  I  u,  v  e  V}. 

Section  3.1:  Storing  Graphs  (Adjacency  Matrix) 

To  store  a  graph,  two  methods  are  common: 

•  Adjacency  Matrix 

•  Adjacency  List 

An  adjacency  matrix  is  a  square  matrix  used  to  represent  a  finite  graph.  The  elements  of  the  matrix  indicate 
whether  pairs  of  vertices  are  adjacent  or  not  in  the  graph. 

Adjacent  means  'next  to  or  adjoining  something  else'  or  to  be  beside  something.  For  example,  your  neighbors  are 
adjacent  to  you.  In  graph  theory,  if  we  can  go  to  node  B  from  node  A,  we  can  say  that  node  B  is  adjacent  to  node 
A.  Now  we  will  learn  about  how  to  store  which  nodes  are  adjacent  to  which  one  via  Adjacency  Matrix.  This  means, 
we  will  represent  which  nodes  share  edge  between  them.  Here  matrix  means  2D  array. 


Here  you  can  see  a  table  beside  the  graph,  this  is  our  adjacency  matrix.  Here  Matrix[i]|J]  =  1  represents  there  is  an 
edge  between  i  and  j.  If  there's  no  edge,  we  simply  put  Matrix[i][j]  =  0. 

These  edges  can  be  weighted,  like  it  can  represent  the  distance  between  two  cities.  Then  we'll  put  the  value  in 
Matrix[i][j]  instead  of  putting  1 . 

The  graph  described  above  is  Bidirectional  or  Undirected,  that  means,  if  we  can  go  to  node  1  from  node  2,  we  can 
also  go  to  node  2  from  node  1.  If  the  graph  was  Directed,  then  there  would've  been  arrow  sign  on  one  side  of  the 
graph.  Even  then,  we  could  represent  it  using  adjacency  matrix. 
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1 


4 


2 


3 


1 

Inf 

3 

7 

5 

2 

3 

Inf 

Inf 

Inf 

3 

Inf 

2 

Inf 

4 

4 

Inf 

Inf 

Inf 

Inf 

We  represent  the  nodes  that  don't  share  edge  by  infinity.  One  thing  to  be  noticed  is  that,  if  the  graph  is  undirected, 
the  matrix  becomes  symmetric. 

The  pseudo-code  to  create  the  matrix: 

Procedure  AdjacencyMatrix(N) :  //N  represents  the  number  of  nodes 

Matrix[N] [N] 
for  i  f  rom  1  to  N 

for  j  f  rom  1  to  N 

Take  input  ->  Mat rix [ ± ] [ j ] 
endfor 
endfor 

We  can  also  populate  the  Matrix  using  this  common  way: 

Procedure  AdjacencyMatrix(N,  E)  :  //  N  ->  number  of  nodes 

Matrix[N][E  //  E  ->  number  of  edges 

for  i  f  rom  1  to  E 

input  ->  nl ,  n2 ,  cost 
Matrix[n1 ] [n2]  =  cost 
Matrix[n2] [nl ]  =  cost 
endfor 

For  directed  graphs,  we  can  remove  Matrix[n2][n1]  =  cost  line. 

The  drawbacks  of  using  Adjacency  Matrix: 

Memory  is  a  huge  problem.  No  matter  how  many  edges  are  there,  we  will  always  need  N  *  N  sized  matrix  where  N 
is  the  number  of  nodes.  If  there  are  1 0000  nodes,  the  matrix  size  will  be  4  *  10000  *  1 0000  around  381  megabytes. 
This  is  a  huge  waste  of  memory  if  we  consider  graphs  that  have  a  few  edges. 

Suppose  we  want  to  find  out  to  which  node  we  can  go  from  a  node  u.  We'll  need  to  check  the  whole  row  of  u,  which 
costs  a  lot  of  time. 

The  only  benefit  is  that,  we  can  easily  find  the  connection  between  u-v  nodes,  and  their  cost  using  Adjacency 
Matrix. 

Java  code  implemented  using  above  pseudo-code: 

import  java .util. Scanner; 

public  class  Represent_Graph_Ad jacency_Matrix 

{ 

private  final  int  vertices 
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private  int[][]  adjacency_matrix; 

public  Represent_Graph_Adjacency_Matrix(int  v) 

{ 

vertices  =  v; 

adjacency_matrix  =  new  int[vertices  +  1] [vertices  +1]; 

} 

public  void  makeEdge(int  to,  int  from,  int  edge) 

{ 

try 

{ 

adjacency_matrix[to  [from]  =  edge 

} 

catch  ( ArraylndexOutOfBoundsException  index) 

{ 

System. out. println( "The  vertices  does  not  exists"); 

} 

} 

public  int  getEdge(int  to,  int  from) 

{ 

try 

{ 

return  adjacency_matrix[to] [from] ; 

} 

catch  (ArraylndexOutOfBoundsException  index) 

{ 

System. out. println( "The  vertices  does  not  exists"); 

} 

return  -1 ; 

} 

public  static  void  main(String  args[]) 

{ 

int  v,  e,  count  =1,  to  =  0,  from  =  0; 

Scanner  sc  =  new  Scanner(System . in) ; 

Represent_Graph_Adjacency_Matrix  graph 

try 

{ 

System. out. println( "Enter  the  number  of  vertices:  "); 
v  =  sc . nextlnt( )  ; 

System. out. println( "Enter  the  number  of  edges:  "); 
e  =  sc . nextlnt ( )  ; 

graph  =  new  Represent_Graph_Adjacency_Matrix(v)  ; 

System. out. println(  "Enter  the  edges:  <to>  <from>"); 
while  (count  <=  e) 

{ 

to  =  sc.nextlnt() ; 
from  =  sc . nextlnt () ; 

graph .makeEdge(to,  from,  1); 
count++ ; 

} 

System. out. println( "The  adjacency  matrix  for  the  given  graph  is:  "); 

System. out . print( "  "); 

for  (int  i  =  1;  i  <=  v  i++) 

System. out. print(i  +  "  "); 

System. out . println( ) ; 
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for  (int  i  =  1 ;  i  <=  v  i++) 

{ 

System. out. print(i  +  "  "); 
for  (int  j  =  1 ;  j  <=  v:  j++) 

System. out. print(graph.getEdge(i,  j)  +  "  "); 

System. out . print ln( ) ; 

} 

} 

catch  (Exception  E) 

{ 

System. out . println( "Somthing  went  wrong"); 

} 

sc . close( )  ; 

} 

} 

Running  the  code:  Save  the  file  and  compile  using  javac  Represent_Graph_Adjacency_Matrix.  java 
Example: 


$  java  Represent_Graph_Adjacency_Matrix 
Enter  the  number  of  vertices: 

4 

Enter  the  number  of  edges: 

6 

Enter  the  edges: 

1  1 
3  4 
2  3 

1  4 

2  4 
1  2 

The  adjacency  matrix  for  the  given  graph  is: 
12  3  4 
1110  1 
2  0  0  1  1 

3  0  0  0  1 

4  0  0  0  0 


Section  3.2:  Introduction  To  Graph  Theory 

Graph  Theory  is  the  study  of  graphs,  which  are  mathematical  structures  used  to  model  pairwise  relations  between 
objects. 

Did  you  know,  almost  all  the  problems  of  planet  Earth  can  be  converted  into  problems  of  Roads  and  Cities,  and 
solved?  Graph  Theory  was  invented  many  years  ago,  even  before  the  invention  of  computer.  Leonhard  Euler  wrote 
a  paper  on  the  Seven  Bridges  of  Konigsberg  which  is  regarded  as  the  first  paper  of  Graph  Theory.  Since  then, 
people  have  come  to  realize  that  if  we  can  convert  any  problem  to  this  City-Road  problem,  we  can  solve  it  easily  by 
Graph  Theory. 

Graph  Theory  has  many  applications. One  of  the  most  common  application  is  to  find  the  shortest  distance  between 
one  city  to  another.  We  all  know  that  to  reach  your  PC,  this  web-page  had  to  travel  many  routers  from  the  server. 
Graph  Theory  helps  it  to  find  out  the  routers  that  needed  to  be  crossed.  During  war,  which  street  needs  to  be 
bombarded  to  disconnect  the  capital  city  from  others,  that  too  can  be  found  out  using  Graph  Theory. 
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Let  us  first  learn  some  basic  definitions  on  Graph  Theory. 

Graph: 

Let's  say,  we  have  6  cities.  We  mark  them  as  1 , 2,  3,  4,  5,  6.  Now  we  connect  the  cities  that  have  roads  between  each 
other. 


This  is  a  simple  graph  where  some  cities  are  shown  with  the  roads  that  are  connecting  them.  In  Graph  Theory,  we 
call  each  of  these  cities  Node  or  Vertex  and  the  roads  are  called  Edge.  Graph  is  simply  a  connection  of  these  nodes 
and  edges. 

A  node  can  represent  a  lot  of  things.  In  some  graphs,  nodes  represent  cities,  some  represent  airports,  some 
represent  a  square  in  a  chessboard.  Edge  represents  the  relation  between  each  nodes.  That  relation  can  be  the 
time  to  go  from  one  airport  to  another,  the  moves  of  a  knight  from  one  square  to  all  the  other  squares  etc. 
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Path  of  Knight  in  a  Chessboard 


In  simple  words,  a  Node  represents  any  object  and  Edge  represents  the  relation  between  two  objects. 

Adjacent  Node: 

If  a  node  A  shares  an  edge  with  node  B,  then  B  is  considered  to  be  adjacent  to  A.  In  other  words,  if  two  nodes  are 
directly  connected,  they  are  called  adjacent  nodes.  One  node  can  have  multiple  adjacent  nodes. 

Directed  and  Undirected  Graph: 

In  directed  graphs,  the  edges  have  direction  signs  on  one  side,  that  means  the  edges  are  Unidirectional.  On  the 
other  hand,  the  edges  of  undirected  graphs  have  direction  signs  on  both  sides,  that  means  they  are  Bidirectional. 
Usually  undirected  graphs  are  represented  with  no  signs  on  the  either  sides  of  the  edges. 

Let's  assume  there  is  a  party  going  on.  The  people  in  the  party  are  represented  by  nodes  and  there  is  an  edge 
between  two  people  if  they  shake  hands.  Then  this  graph  is  undirected  because  any  person  A  shake  hands  with 
person  B  if  and  only  if  B  also  shakes  hands  with  A.  In  contrast,  if  the  edges  from  a  person  A  to  another  person  B 
corresponds  to  A's  admiring  B,  then  this  graph  is  directed,  because  admiration  is  not  necessarily  reciprocated.  The 
former  type  of  graph  is  called  an  undirected  graph  and  the  edges  are  called  undirected  edges  while  the  latter  type  of 
graph  is  called  a  directed  graph  and  the  edges  are  called  directed  edges. 

Weighted  and  Unweighted  Graph: 

A  weighted  graph  is  a  graph  in  which  a  number  (the  weight)  is  assigned  to  each  edge.  Such  weights  might  represent 
for  example  costs,  lengths  or  capacities,  depending  on  the  problem  at  hand. 
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An  unweighted  graph  is  simply  the  opposite.  We  assume  that,  the  weight  of  all  the  edges  are  same  (presumably  1 ). 

Path: 

A  path  represents  a  way  of  going  from  one  node  to  another.  It  consists  of  sequence  of  edges.  There  can  be  multiple 


paths  between  two  nodes. 

In  the  example  above,  there  are  two  paths  from  A  to  D.  A->B,  B->C,  C->D  is  one  path.  The  cost  of  this  path  is  3  +  4  + 
2  =  9.  Again,  there's  another  path  A->D.  The  cost  of  this  path  is  10.  The  path  that  costs  the  lowest  is  called  shortest 
path. 

Degree: 

The  degree  of  a  vertex  is  the  number  of  edges  that  are  connected  to  it.  If  there's  any  edge  that  connects  to  the 
vertex  at  both  ends  (a  loop)  is  counted  twice. 


GoalKicker.com  -  Algorithms  Notes  for  Professionals 


14 


In  directed  graphs,  the  nodes  have  two  types  of  degrees: 

•  In-degree:  The  number  of  edges  that  point  to  the  node. 

•  Out-degree:  The  number  of  edges  that  point  from  the  node  to  other  nodes. 

For  undirected  graphs,  they  are  simply  called  degree. 


Some  Algorithms  Related  to  Graph  Theory 

•  Bellman-Ford  algorithm 

•  Dijkstra's  algorithm 

•  Ford-Fulkerson  algorithm 

•  Kruskal's  algorithm 

•  Nearest  neighbour  algorithm 

•  Prim's  algorithm 

•  Depth-first  search 

•  Breadth-first  search 


Section  3.3:  Storing  Graphs  (Adjacency  List) 

Adjacency  list  is  a  collection  of  unordered  lists  used  to  represent  a  finite  graph.  Each  list  describes  the  set  of 
neighbors  of  a  vertex  in  a  graph.  It  takes  less  memory  to  store  graphs. 


Let's  see  a  graph,  and  its  adjacency  matrix: 


Now  we  create  a  list  using  these  values. 
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This  is  called  adjacency  list.  It  shows  which  nodes  are  connected  to  which  nodes.  We  can  store  this  information 
using  a  2D  array.  But  will  cost  us  the  same  memory  as  Adjacency  Matrix.  Instead  we  are  going  to  use  dynamically 
allocated  memory  to  store  this  one. 

Many  languages  support  Vector  or  List  which  we  can  use  to  store  adjacency  list.  For  these,  we  don't  need  to  specify 
the  size  of  the  List.  We  only  need  to  specify  the  maximum  number  of  nodes. 

The  pseudo-code  will  be: 

Procedure  Adjacency-List(maxN,  E) 
edgejmaxN]  =  Vector() 
for  i  f  rom  1  to  E 
input  ->  x,  y 
edgejx] .push(y) 
edgejy] .push(x) 
end  for 
Return  edge 

Since  this  one  is  an  undirected  graph,  it  there  is  an  edge  from  x  to  y,  there  is  also  an  edge  from  y  to  x.  If  it  was  a 
directed  graph,  we'd  omit  the  second  one.  For  weighted  graphs,  we  need  to  store  the  cost  too.  We'll  create  another 
vector  or  list  named  cost[]  to  store  these.  The  pseudo-code: 

Procedure  Adjacency-List(maxN,  E)  : 
edgejmaxN]  =  Vectorj) 
costjmaxN]  =  Vector() 
for  i  f  rom  1  to  E 

input  ->  x,  y,  w 
edgejx] .push(y) 
costjx] .push(w) 
end  for 

Return  edge,  cost 

From  this  one,  we  can  easily  find  out  the  total  number  of  nodes  connected  to  any  node,  and  what  these  nodes  are. 


//  maxN  denotes  the  maximum  number  of  nodes 
//  E  denotes  the  number  of  edges 

//  Here  x,  y  denotes  there  is  an  edge  between  x,  y 
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It  takes  less  time  than  Adjacency  Matrix.  But  if  we  needed  to  find  out  if  there's  an  edge  between  u  and  v,  it'd  have 
been  easier  if  we  kept  an  adjacency  matrix. 

Section  3.4:  Topological  Sort 

A  topological  ordering,  or  a  topological  sort,  orders  the  vertices  in  a  directed  acyclic  graph  on  a  line,  i.e.  in  a  list, 
such  that  all  directed  edges  go  from  left  to  right.  Such  an  ordering  cannot  exist  if  the  graph  contains  a  directed  cycle 
because  there  is  no  way  that  you  can  keep  going  right  on  a  line  and  still  return  back  to  where  you  started  from. 

Formally,  in  a  graph  G  =  (V,  E),  then  a  linear  ordering  of  all  itsvertices  is  such  that  if  G  contains  an  edge  (u,  v)  e 
Efrom  vertex  u  to  vertex  v  then  u  precedes  v  in  the  ordering. 

It  is  important  to  note  that  each  DAG  has  at  least  one  topological  sort. 

There  are  known  algorithms  for  constructing  a  topological  ordering  of  any  DAG  in  linear  time,  one  example  is: 

1 .  Call  depth_first_search(G)  to  compute  finishing  times  v.  f  for  each  vertex  v 

2.  As  each  vertex  is  finished,  insert  it  into  the  front  of  a  linked  list 

3.  the  linked  list  of  vertices,  as  it  is  now  sorted. 

A  topological  sort  can  be  performed  in  □  (V  +  E)  time,  since  the  depth-first  search  algorithm  takes  IH(V  +  E)  time 
and  it  takes  0(1)  (constant  time)  to  insert  each  of  |  V  |  vertices  into  the  front  of  a  linked  list. 

Many  applications  use  directed  acyclic  graphs  to  indicate  precedences  among  events.  We  use  topological  sorting  so 
that  we  get  an  ordering  to  process  each  vertex  before  any  of  its  successors. 

Vertices  in  a  graph  may  represent  tasks  to  be  performed  and  the  edges  may  represent  constraints  that  one  task 
must  be  performed  before  another;  a  topological  ordering  is  a  valid  sequence  to  perform  the  tasks  set  of  tasks 
described  in  V. 

Problem  instance  and  its  solution 

Let  a  vertice  v  describe  a  Task(hours_to_complete  int),  i.  e.  Task(4)  describes  a  Task  that  takes  4  hours  to 
complete,  and  an  edge  e  describe  a  Cooldown(hours  :  int)  such  that  Cooldown(3)  describes  a  duration  of  time  to 
cool  down  after  a  completed  task. 

Let  our  graph  be  called  dag  (since  it  is  a  directed  acyclic  graph),  and  let  it  contain  5  vertices: 

A  <-  dag . add_vertex(Task(4) ) ; 

B  <-  dag . add_vertex(Task( 5) ) ; 

C  <-  dag . add_vertex(Task(3) ) ; 

D  <-  dag . add_vertex(Task(2) )  ; 

E  <-  dag . add_vertex(Task(7) ) ; 

where  we  connect  the  vertices  with  directed  edges  such  that  the  graph  is  acyclic, 


//  A  --->  C  --- 

//  /  / 

//  v  v 

//  B  --->  D  --> 
dag . add_edge( A, 
dag . add_edge( A, 
dag .add_edge(B, 
dag . add_edge(C, 
dag . add_edge(C, 
dag .add_edge(D, 


/ 

v 

E 

B,  Cooldown(2)) 

C,  Cooldown(2)) 

D,  Cooldown(l)) 

D,  Cooldown(l)) 

E,  Cooldown(l)) 
E,  Cooldown(3)) 
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then  there  are  three  possible  topological  orderings  between  A  and  E, 

1.  A  ->  B  ->  D  ->  E 

2.  A  ->  C  ->  D  ->  E 

3.  A  ->  C  ->  E 

Section  3.5:  Detecting  a  cycle  in  a  directed  graph  using  Depth 
First  Traversal 

A  cycle  in  a  directed  graph  exists  if  there's  a  back  edge  discovered  during  a  DFS.  A  back  edge  is  an  edge  from  a  node 
to  itself  or  one  of  the  ancestors  in  a  DFS  tree.  For  a  disconnected  graph,  we  get  a  DFS  forest,  so  you  have  to  iterate 
through  all  vertices  in  the  graph  to  find  disjoint  DFS  trees. 

C++  implementation: 

#include  <iostream> 

#include  <list> 

using  namespace  std ; 

#define  NUM_V  4 

bool  helper(list<int>  *graph,  int  u,  bool*  visited,  bool*  recStack) 

{ 

visited [u ]=true ; 
recStack[u]=true; 
list<int> :: iterator  i 

for(i  =  graph [u ]. begin( )  i ! =graph [u ] . end( ) ;++i) 

{ 

if ( recStack[*i] )  //if  vertice  v  is  found  in  recursion  stack  of  this  DFS  traversal 
return  true; 

else  if(*i==u)  //if  there's  an  edge  from  the  vertex  to  itself 
return  true; 
else  if (! visited [ *i  ] ) 

{  if (helper(graph,  *i,  visited,  recStack)) 

return  true; 

} 

} 

recStack [ u ] =f alse ; 
return  false; 

} 

/* 

/The  wrapper  function  calls  helper  function  on  each  vertices  which  have  not  been  visited.  Helper 
function  returns  true  if  it  detects  a  back  edge  in  the  subgraph(tree)  or  false. 

*/ 

bool  isCyclic(list<int>  *graph,  int  V) 

{ 

bool  visited[V];  //array  to  track  vertices  already  visited 

bool  recStack[V];  //array  to  track  vertices  in  recursion  stack  of  the  traversal. 
for(int  i  =  0  i<V  i++) 

visited[i]=false,  recStack[i]=false;  //initialize  all  vertices  as  not  visited  and  not 
recursed 

for(int  u  =  0  u  <  V  u++)  //Iteratively  checks  if  every  vertices  have  been  visited 
{  if (visited [ u ] ==false) 

{  if (helper(graph,  u,  visited,  recStack))  //checks  if  the  DFS  tree  from  the  vertex 
contains  a  cycle 

return  true; 
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} 


} 

return  false; 

} 

/* 

Driver  function 
*/ 

int  main() 

{ 

list<int>*  graph  =  new  list<int>[NUM_V] ; 

graph [0] . push_back( 1 ) ; 

graph[0] . push_back(2) ; 

graph[1  ] . push_back(2)  ; 

graph[2] . push_back(0)  ; 

graph[2] . push_back(3)  ; 

graph[3] . push_back(3)  ; 

bool  res  =  isCyclic(graph,  NUM_V)  ; 

cout<<res<<endl ; 

} 

Result:  As  shown  below,  there  are  three  back  edges  in  the  graph.  One  between  vertex  0  and  2;  between  vertice  0, 1 , 
and  2;  and  vertex  3.  Time  complexity  of  search  is  0(V+E)  where  V  is  the  number  of  vertices  and  E  is  the  number  of 
edges. 


Section  3.6:  Thorup's  algorithm 

Thorup's  algorithm  for  single  source  shortest  path  for  undirected  graph  has  the  time  complexity  O(m),  lower  than 
Dijkstra. 

Basic  ideas  are  the  following.  (Sorry,  I  didn't  try  implementing  it  yet,  so  I  might  miss  some  minor  details.  And  the 
original  paper  is  paywalled  so  I  tried  to  reconstruct  it  from  other  sources  referencing  it.  Please  remove  this 
comment  if  you  could  verify.) 

•  There  are  ways  to  find  the  spanning  tree  in  O(m)  (not  described  here).  You  need  to  "grow"  the  spanning  tree 
from  the  shortest  edge  to  the  longest,  and  it  would  be  a  forest  with  several  connected  components  before 
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fully  grown. 

•  Select  an  integer  b  (b>=2)  and  only  consider  the  spanning  forests  with  length  limit  bAk.  Merge  the 
components  which  are  exactly  the  same  but  with  different  k,  and  call  the  minimum  k  the  level  of  the 
component.  Then  logically  make  components  into  a  tree,  u  is  the  parent  of  v  iff  u  is  the  smallest  component 
distinct  from  v  that  fully  contains  v.  The  root  is  the  whole  graph  and  the  leaves  are  single  vertices  in  the 
original  graph  (with  the  level  of  negative  infinity).  The  tree  still  has  only  O(n)  nodes. 

•  Maintain  the  distance  of  each  component  to  the  source  (like  in  Dijkstra's  algorithm).  The  distance  of  a 
component  with  more  than  one  vertices  is  the  minimum  distance  of  its  unexpanded  children.  Set  the 
distance  of  the  source  vertex  to  0  and  update  the  ancestors  accordingly. 

•  Consider  the  distances  in  base  b.  When  visiting  a  node  in  level  k  the  first  time,  put  its  children  into  buckets 
shared  by  all  nodes  of  level  k  (as  in  bucket  sort,  replacing  the  heap  in  Dijkstra's  algorithm)  by  the  digit  k  and 
higher  of  its  distance.  Each  time  visiting  a  node,  consider  only  its  first  b  buckets,  visit  and  remove  each  of 
them,  update  the  distance  of  the  current  node,  and  relink  the  current  node  to  its  own  parent  using  the  new 
distance  and  wait  for  the  next  visit  for  the  following  buckets. 

•  When  a  leaf  is  visited,  the  current  distance  is  the  final  distance  of  the  vertex.  Expand  all  edges  from  it  in  the 
original  graph  and  update  the  distances  accordingly. 

•  Visit  the  root  node  (whole  graph)  repeatedly  until  the  destination  is  reached. 

It  is  based  on  the  fact  that,  there  isn't  an  edge  with  length  less  than  I  between  two  connected  components  of  the 
spanning  forest  with  length  limitation  I,  so,  starting  at  distance  x,  you  could  focus  only  on  one  connected 
component  until  you  reach  the  distance  x  +  I.  You'll  visit  some  vertices  before  vertices  with  shorter  distance  are  all 
visited,  but  that  doesn't  matter  because  it  is  known  there  won't  be  a  shorter  path  to  here  from  those  vertices.  Other 
parts  work  like  the  bucket  sort  /  MSD  radix  sort,  and  of  course,  it  requires  the  O(m)  spanning  tree. 


GoalKicker.com  -  Algorithms  Notes  for  Professionals 


20 


Chapter  4:  Graph  Traversals 

Section  4.1:  Depth  First  Search  traversal  function 

The  function  takes  the  argument  of  the  current  node  index,  adjacency  list  (stored  in  vector  of  vectors  in  this 
example),  and  vector  of  boolean  to  keep  track  of  which  node  has  been  visited. 

void  dfs(int  node,  vector<vector<int>>*  graph,  vector<bool>*  visited)  { 

//  check  whether  node  has  been  visited  before 
if ( (*visited) [node  ) 

return 

//  set  as  visited  to  avoid  visiting  the  same  node  twice 
(*visited) [node]  =  true 

//  perform  some  action  here 
cout  <<  node 

//  traverse  to  the  adjacent  nodes  in  depth- first  manner 
for(int  i  =  0;  i  <  (*graph) [node] .size( ) ;  ++i) 
dfs((*graph) [node  [i  ,  graph,  visited); 

} 
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Chapter  5:  Dijkstra’s  Algorithm 

Section  5.1:  Dijkstra's  Shortest  Path  Algorithm 

Before  proceeding,  it  is  recommended  to  have  a  brief  idea  about  Adjacency  Matrix  and  BFS 

Dijkstra's  algorithm  is  known  as  single-source  shortest  path  algorithm.  It  is  used  for  finding  the  shortest  paths 
between  nodes  in  a  graph,  which  may  represent,  for  example,  road  networks.  It  was  conceived  by  Edsger  W. 

Dijkstra  in  1 956  and  published  three  years  later. 

We  can  find  shortest  path  using  Breadth  First  Search  (BFS)  searching  algorithm.  This  algorithm  works  fine,  but  the 
problem  is,  it  assumes  the  cost  of  traversing  each  path  is  same,  that  means  the  cost  of  each  edge  is  same.  Dijkstra's 
algorithm  helps  us  to  find  the  shortest  path  where  the  cost  of  each  path  is  not  the  same. 

At  first  we  will  see,  how  to  modify  BFS  to  write  Dijkstra's  algorithm,  then  we  will  add  priority  queue  to  make  it  a 
complete  Dijkstra's  algorithm. 

Let's  say,  the  distance  of  each  node  from  the  source  is  kept  in  d[]  array.  As  in,  d[3]  represents  that  d[3]  time  is  taken 
to  reach  node  3  from  source.  If  we  don't  know  the  distance,  we  will  store  infinity  in  d[3].  Also,  let  cost[u][v] 
represent  the  cost  of  u-v.  That  means  it  takes  cost[u][v]  to  go  from  u  node  to  v  node. 


We  need  to  understand  Edge  Relaxation.  Let's  say,  from  your  house,  that  is  source,  it  takes  10  minutes  to  go  to 
place  A.  And  it  takes  25  minutes  to  go  to  place  B.  We  have, 

d [A]  =  10 
d[B]  =  25 

Now  let's  say  it  takes  7  minutes  to  go  from  place  A  to  place  B,  that  means: 
costjA] [B]  =7 

Then  we  can  go  to  place  B  from  source  by  going  to  place  A  from  source  and  then  from  place  A,  going  to  place  B, 
which  will  take  10  +  7=17  minutes,  instead  of  25  minutes.  So, 

d [ A ]  +  cost[A][B]  <  d[B] 

Then  we  update, 

d[B]  =  d [ A ]  +  cost[A][B] 

This  is  called  relaxation.  We  will  go  from  node  u  to  node  v  and  if  d[u]  +  cost[u][v]  <  d[v]  then  we  will  update  d[v]  = 
d[u]  +  cost[u][v]. 

In  BFS,  we  didn't  need  to  visit  any  node  twice.  We  only  checked  if  a  node  is  visited  or  not.  If  it  was  not  visited,  we 
pushed  the  node  in  queue,  marked  it  as  visited  and  incremented  the  distance  by  1 .  In  Dijkstra,  we  can  push  a  node 
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in  queue  and  instead  of  updating  it  with  visited,  we  relax  or  update  the  new  edge.  Let's  look  at  one  example: 


Let's  assume,  Node  1  is  the  Source.  Then, 

d  [  1  ]  =  0 

d [ 2 ]  =  d [ 3 ]  =  d[4]  =  infinity  (or  a  large  value) 

We  set,  d[2],  d[3]  and  d[4]  to  infinity  because  we  don't  know  the  distance  yet.  And  the  distance  of  source  is  of 
course  0.  Now,  we  go  to  other  nodes  from  source  and  if  we  can  update  them,  then  we'll  push  them  in  the  queue. 
Say  for  example,  we'll  traverse  edge  1-2.  As  d[1]  +  2  <  d[2]  which  will  make  d[2]  =  2.  Similarly,  we'll  traverse  edge  1-3 
which  makes  d  [3]  =  5. 

We  can  clearly  see  that  5  is  not  the  shortest  distance  we  can  cross  to  go  to  node  3.  So  traversing  a  node  only  once, 
like  BFS,  doesn't  work  here.  If  we  go  from  node  2  to  node  3  using  edge  2-3,  we  can  update  d[3]  =  d[2]  +  1  =  3.  So  we 
can  see  that  one  node  can  be  updated  many  times.  How  many  times  you  ask?  The  maximum  number  of  times  a 
node  can  be  updated  is  the  number  of  in-degree  of  a  node. 

Let's  see  the  pseudo-code  for  visiting  any  node  multiple  times.  We  will  simply  modify  BFS: 

procedure  BFSmodified(G,  source) : 

Q  =  queue() 
distance!]  =  infinity 
Q.enqueue(source) 
distance] source ]=0 
while  Q  is  not  empty 
u  <-  Q.pop() 

for  all  edges  from  u  to  v  in  G.adjacentEdges(v)  do 
if  distance[u  +  cost[u  [v]  <  distance[v] 
distance[v]  =  distance[u]  +  cost[u][v] 
end  if 
end  for 
end  while 
Return  distance 
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This  can  be  used  to  find  the  shortest  path  of  all  node  from  the  source.  The  complexity  of  this  code  is  not  so  good. 
Here's  why, 

In  BFS,  when  we  go  from  node  1  to  all  other  nodes,  we  follow  first  come,  first  serve  method.  For  example,  we  went  to 
node  3  from  source  before  processing  node  2.  If  we  go  to  node  3  from  source,  we  update  node  4  as  5  +  3  =  8. 
When  we  again  update  node  3  from  node  2,  we  need  to  update  node  4  as  3  +  3  =  6  again!  So  node  4  is  updated 
twice. 

Dijkstra  proposed,  instead  of  going  for  First  come,  first  serve  method,  if  we  update  the  nearest  nodes  first,  then  it'll 
take  less  updates.  If  we  processed  node  2  before,  then  node  3  would  have  been  updated  before,  and  after 
updating  node  4  accordingly,  we'd  easily  get  the  shortest  distance!  The  idea  is  to  choose  from  the  queue,  the  node, 
that  is  closest  to  the  source.  So  we  will  use  Priority  Queue  here  so  that  when  we  pop  the  queue,  it  will  bring  us  the 
closest  node  u  from  source.  How  will  it  do  that?  It'll  check  the  value  of  d[u]  with  it. 

Let's  see  the  pseudo-code: 

procedure  dij kstra(G,  source) : 

Q  =  priority_queue( ) 
distance!]  =  infinity 
Q.enqueue(source) 
distance[source  =  0 
while  Q  is  not  empty 

u  <-  nodes  in  Q  with  minimum  distance]] 
remove  u  from  the  Q 

for  all  edges  from  u  to  v  in  G.adjacentEdges(v)  do 
if  distance[u  +  cost[u  [v]  <  distance[v] 
distance[v]  =  distance[u]  +  cost[u][v] 

Q.enqueue(v) 
end  if 
end  for 
end  while 
Return  distance 

The  pseudo-code  returns  distance  of  all  other  nodes  from  the  source.  If  we  want  to  know  distance  of  a  single  node 
v,  we  can  simply  return  the  value  when  v  is  popped  from  the  queue. 

Now,  does  Dijkstra's  Algorithm  work  when  there's  a  negative  edge?  If  there's  a  negative  cycle,  then  infinity  loop  will 
occur,  as  it  will  keep  reducing  the  cost  every  time.  Even  if  there  is  a  negative  edge,  Dijkstra  won't  work,  unless  we 
return  right  after  the  target  is  popped.  But  then,  it  won't  be  a  Dijkstra  algorithm.  We'll  need  Bellman-Ford  algorithm 
for  processing  negative  edge/cycle. 

Complexity: 

The  complexity  of  BFS  is  0(log(V+E))  where  V  is  the  number  of  nodes  and  E  is  the  number  of  edges.  For  Dijkstra, 
the  complexity  is  similar,  but  sorting  of  Priority  Queue  takes  O(logV).  So  the  total  complexity  is:  0(Vlog(V)+E) 

Below  is  a  Java  example  to  solve  Dijkstra's  Shortest  Path  Algorithm  using  Adjacency  Matrix 

import  java. util.*; 
import  java.lang.*; 
import  java.io.*; 

class  ShortestPath 

{ 

static  final  int  V=9; 

int  minDistance(int  dist]  ,  Boolean  sptSet]]) 

{ 
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int  min  =  Integer .MAX_VALUE,  min_index=-1 ; 


for  (int  v  =  0;  v  <  V  v++) 

if  (sptSet[v]  ==  false  &&  dist[v]  <=  min) 

{ 

min  =  dist[v] ; 
min_index  =  v 

} 

return  min_index; 

} 

void  printSolution(int  dist[  ,  int  n) 

{ 

System. out . println( "Vertex  Distance  from  Source"); 
for  (int  i  =  0;  i  <  V  i++) 

System. out. println(i+"  \t\t  " +d is t [ i ] ) ; 

} 

void  dijkstra(int  graph[][],  int  src) 

{ 


Boolean  sptSet[]  =  new  Boolean[V 

for  (int  i  =  0;  i  <  V  i++) 

{ 

dist [ i ]  =  Integer . MAX_VALUE ; 
sptSet[i  =  false 

} 

dist[src]  =  0; 

for  (int  count  =  0;  count  <  V  1 ;  count++) 

{ 

int  u  =  minDistance(dist,  sptSet) ; 
sptSet[u  =  true 

for  (int  v  =  0;  v  <  V  v++) 

if  (!sptSet[v]  &&  graph[u] [v] !=0  && 

dist [ u ]  ! =  Integer. MAX_ VALUE  && 
dist[u  +graph[u][v]  <  dist[v]) 
dist[v]  =  dist[u  +  graph[u  [v] ; 

} 

printSolution(dist,  V); 

} 


public  static  void  main 

{ 

int  graph[][]  =  new  int] 

(String 

□ 

args 

0 

: ] t ] {{0. 

4, 

0, 

0, 

0, 

0,  0, 

8, 

0}, 

{4, 

0, 

8, 

0, 

0, 

0,  0, 

11 , 

0}, 

{0, 

8, 

0, 

7, 

0, 

4,  0, 

0, 

2}, 

{0, 

0, 

7, 

0, 

9, 

14,  0, 

0, 

0}, 

{0, 

0, 

0, 

9, 

0, 

10,  0, 

0, 

0}, 

{0, 

0, 

4, 

14, 

IE 

1,  0,  2 

, 

,  0} 

{0, 

0, 

0, 

0, 

0, 

2,  0, 

1  , 

6}, 

{8, 

11 

,  0, 

0, 

0, 

0,  1, 

0, 

7}, 

{0, 

0, 

2, 

0, 

0, 

0,  6, 

7, 

0} 

}  ; 

ShortestPath  t  =  new  ShortestPath( ) ; 
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} 


t .dijkstra(graph,  0); 


Expected  output  of  the  program  is 


Vertex 

0 

1 

2 

3 

4 

5 

6 

7 

8 


Distance  from  Source 

0 

4 

12 

19 

21 

11 

9 

8 

14 
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Chapter  6:  A*  Pathfinding 

Section  6.1:  Introduction  to  A* 

A*  (A  star)  is  a  search  algorithm  that  is  used  for  finding  path  from  one  node  to  another.  So  it  can  be  compared  with 
Breadth  First  Search,  or  Dijkstra’s  algorithm,  or  Depth  First  Search,  or  Best  First  Search.  A*  algorithm  is  widely  used 
in  graph  search  for  being  better  in  efficiency  and  accuracy,  where  graph  pre-processing  is  not  an  option. 

A*  is  a  an  specialization  of  Best  First  Search  ,  in  which  the  function  of  evaluation /is  define  in  a  particular  way. 

f(n)  =g(n)  +  h(n)  is  the  minimum  cost  since  the  initial  node  to  the  objectives  conditioned  to  go  thought  node  n. 

g(n)  is  the  minimum  cost  from  the  initial  node  to  n. 

h(n)  is  the  minimum  cost  from  n  to  the  closest  objective  to  n 

A*  is  an  informed  search  algorithm  and  it  always  guarantees  to  find  the  smallest  path  (path  with  minimum  cost)  in 
the  least  possible  time  (if  uses  admissible  heuristic).  So  it  is  both  complete  and  optimal.  The  following  animation 
demonstrates  A*  search- 


* 


* 

Section  6.2:  A*  Pathfinding  through  a  maze  with  no  obstacles 

Let's  say  we  have  the  following  4  by  4  grid: 
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Let's  assume  that  this  is  a  maze.  There  are  no  walls/obstacles,  though.  We  only  have  a  starting  point  (the  green 
square),  and  an  ending  point  (the  red  square).  Let's  also  assume  that  in  order  to  get  from  green  to  red,  we  cannot 
move  diagonally.  So,  starting  from  the  green  square,  let's  see  which  squares  we  can  move  to,  and  highlight  them  in 
blue: 
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In  order  to  choose  which  square  to  move  to  next,  we  need  to  take  into  account  2  heuristics: 

1 .  The  "g"  value  -  This  is  how  far  away  this  node  is  from  the  green  square. 

2.  The  "h"  value  -  This  is  how  far  away  this  node  is  from  the  red  square. 

3.  The  "f"  value  -  This  is  the  sum  of  the  "g"  value  and  the  "h"  value.  This  is  the  final  number  which  tells  us  which 
node  to  move  to. 

In  order  to  calculate  these  heuristics,  this  is  the  formula  we  will  use:  distance  =  abs(from.x  -  to.x)  + 
abs(from.y  -  to.y) 

This  is  known  as  the  'Manhattan  Distance"  formula. 

Let's  calculate  the  "g"  value  for  the  blue  square  immediately  to  the  left  of  the  green  square:  abs(3  -  2)  +  abs(2  - 
2)  =  1 

Great!  We've  got  the  value:  1 .  Now,  let's  try  calculating  the  "h"  value:  abs(2  -  0)  +  abs(2  -  0)  =  4 
Perfect.  Now,  let's  get  the  "f"  value:  1+4  =  5 
So,  the  final  value  for  this  node  is  "5". 

Let's  do  the  same  for  all  the  other  blue  squares.  The  big  number  in  the  center  of  each  square  is  the  "f"  value,  while 
the  number  on  the  top  left  is  the  "g"  value,  and  the  number  on  the  top  right  is  the  "h"  value: 
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We've  calculated  the  g,  h,  and  f  values  for  all  of  the  blue  nodes.  Now,  which  do  we  pick? 

Whichever  one  has  the  lowest  f  value. 

However,  in  this  case,  we  have  2  nodes  with  the  same  f  value,  5.  How  do  we  pick  between  them? 

Simply,  either  choose  one  at  random,  or  have  a  priority  set.  I  usually  prefer  to  have  a  priority  like  so:  "Right  >  Up  > 
Down  >  Left" 

One  of  the  nodes  with  the  f  value  of  5  takes  us  in  the  "Down"  direction,  and  the  other  takes  us  "Left".  Since  Down  is 
at  a  higher  priority  than  Left,  we  choose  the  square  which  takes  us  "Down". 

I  now  mark  the  nodes  which  we  calculated  the  heuristics  for,  but  did  not  move  to,  as  orange,  and  the  node  which 
we  chose  as  cyan: 
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Alright,  now  let's  calculate  the  same  heuristics  for  the  nodes  around  the  cyan  node: 


Again,  we  choose  the  node  going  down  from  the  cyan  node,  as  all  the  options  have  the  same  f  value: 
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Let's  calculate  the  heuristics  for  the  only  neighbour  that  the  cyan  node  has: 


Alright,  since  we  will  follow  the  same  pattern  we  have  been  following: 
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Let's  move  there: 
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Finally,  we  can  see  that  we  have  a  winning  square  beside  us,  so  we  move  there,  and  we  are  done. 

Section  6.3:  Solving  8-puzzle  problem  using  A*  algorithm 

Problem  definition: 


An  8  puzzle  is  a  simple  game  consisting  of  a  3  x  3  grid  (containing  9  squares).  One  of  the  squares  is  empty.  The 
object  is  to  move  to  squares  around  into  different  positions  and  having  the  numbers  displayed  in  the  "goal  state". 


1 

2 

3 

8 

4 

7 

6 

5 

Given  an  initial  state  of  8-puzzle  game  and  a  final  state  of  to  be  reached,  find  the  most  cost-effective  path  to  reach 
the  final  state  from  initial  state. 


Initial  state: 


_  1  3 
4  2  5 
7  8  6 
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1  2 
4  5 
7  8 


Final  state: 


3 

6 


Heuristic  to  be  assumed: 


Let  us  consider  the  Manhattan  distance  between  the  current  and  final  state  as  the  heuristic  for  this  problem 
statement. 

h(n)  =  |  x  p  I  +  I  y  q  I 

where  x  and  y  are  cell  co  ordinates  in  the  current  state 
p  and  q  are  cell  co  ordinates  in  the  final  state 


Total  cost  function: 


So  the  total  cost  function  f  (n)  is  given  by, 

f(n)  =  g(n)  +  h(n),  where  g(n)  is  the  cost  required  to  reach  the  current  state  from  given  initial 
state 


Solution  to  example  problem: 


First  we  find  the  heuristic  value  required  to  reach  the  final  state  from  initial  state.  The  cost  function,  g(n)  =  0,  as  we 
are  in  the  initial  state 

h(n)  =  8 

The  above  value  is  obtained,  as  1  in  the  current  state  is  1  horizontal  distance  away  than  the  1  in  final  state.  Same 
goes  for  2,  5,  6.  _  is  2  horizontal  distance  away  and  2  vertical  distance  away.  So  total  value  for  h(n)  is  1  +  1  +  1  +  1  + 
2  +  2  =  8.  Total  cost  function  f  (n)  is  equal  to  8  +  0  =  8. 

Now,  the  possible  states  that  can  be  reached  from  initial  state  are  found  and  it  happens  that  we  can  either  move  _ 
to  right  or  downwards. 

So  states  obtained  after  moving  those  moves  are: 

1  _  3  4  13 

4  2  5  _  2  5 

7  8  6  7  8  6 

(1)  (2) 

Again  the  total  cost  function  is  computed  for  these  states  using  the  method  described  above  and  it  turns  out  to  be 
6  and  7  respectively.  We  chose  the  state  with  minimum  cost  which  is  state  (1).  The  next  possible  moves  can  be  Left, 
Right  or  Down.  We  won't  move  Left  as  we  were  previously  in  that  state.  So,  we  can  move  Right  or  Down. 

Again  we  find  the  states  obtained  from  (1). 

1  3  _  12  3 
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4  2  5  4  _  5 

7  8  6  7  8  6 

(3)  (4) 

(3)  leads  to  cost  function  equal  to  6  and  (4)  leads  to  4.  Also,  we  will  consider  (2)  obtained  before  which  has  cost 
function  equal  to  7.  Choosing  minimum  from  them  leads  to  (4).  Next  possible  moves  can  be  Left  or  Right  or  Down. 
We  get  states: 

1  2  3  1  2  3  1  2  3 

_  4  5  4  5  _  485 

786  786  7  _  6 

(5)  (6)  (7) 

We  get  costs  equal  to  5,  2  and  4  for  (5),  (6)  and  (7)  respectively.  Also,  we  have  previous  states  (3)  and  (2)  with  6  and  7 
respectively.  We  chose  minimum  cost  state  which  is  (6).  Next  possible  moves  are  Up,  and  Down  and  clearly  Down 
will  lead  us  to  final  state  leading  to  heuristic  function  value  equal  to  0. 
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Chapter  7:  A*  Pathfinding  Algorithm 

This  topic  is  going  to  focus  on  the  A*  Pathfinding  algorithm,  how  it's  used,  and  why  it  works. 

Note  to  future  contributors:  I  have  added  an  example  for  A*  Pathfinding  without  any  obstacles,  on  a  4x4  grid.  An 
example  with  obstacles  is  still  needed. 

Section  7.1:  Simple  Example  of  A*  Pathfinding:  A  maze  with  no 
obstacles 


Let's  assume  that  this  is  a  maze.  There  are  no  walls/obstacles,  though.  We  only  have  a  starting  point  (the  green 
square),  and  an  ending  point  (the  red  square).  Let's  also  assume  that  in  order  to  get  from  green  to  red,  we  cannot 
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move  diagonally.  So,  starting  from  the  green  square,  let's  see  which  squares  we  can  move  to,  and  highlight  them  in 


In  order  to  choose  which  square  to  move  to  next,  we  need  to  take  into  account  2  heuristics: 

1 .  The  "g"  value  -  This  is  how  far  away  this  node  is  from  the  green  square. 

2.  The  "h"  value  -  This  is  how  far  away  this  node  is  from  the  red  square. 

3.  The  "f"  value  -  This  is  the  sum  of  the  "g"  value  and  the  "h"  value.  This  is  the  final  number  which  tells  us  which 
node  to  move  to. 

In  order  to  calculate  these  heuristics,  this  is  the  formula  we  will  use:  distance  =  abs(from.x  -  to.x)  + 
abs(from.y  -  to.y) 

This  is  known  as  the  'Manhattan  Distance"  formula. 

Let's  calculate  the  "g"  value  for  the  blue  square  immediately  to  the  left  of  the  green  square:  abs(3  -  2)  +  abs(2  - 
2)  =  1 

Great!  We've  got  the  value:  1 .  Now,  let's  try  calculating  the  "h"  value:  abs(2  -  0)  +  abs(2  -  0)  =  4 
Perfect.  Now,  let's  get  the  "f"  value:  1+4  =  5 
So,  the  final  value  for  this  node  is  "5". 

Let's  do  the  same  for  all  the  other  blue  squares.  The  big  number  in  the  center  of  each  square  is  the  "f"  value,  while 
the  number  on  the  top  left  is  the  "g"  value,  and  the  number  on  the  top  right  is  the  "h"  value: 
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We've  calculated  the  g,  h,  and  f  values  for  all  of  the  blue  nodes.  Now,  which  do  we  pick? 

Whichever  one  has  the  lowest  f  value. 

However,  in  this  case,  we  have  2  nodes  with  the  same  f  value,  5.  How  do  we  pick  between  them? 

Simply,  either  choose  one  at  random,  or  have  a  priority  set.  I  usually  prefer  to  have  a  priority  like  so:  "Right  >  Up  > 
Down  >  Left" 

One  of  the  nodes  with  the  f  value  of  5  takes  us  in  the  "Down"  direction,  and  the  other  takes  us  "Left".  Since  Down  is 
at  a  higher  priority  than  Left,  we  choose  the  square  which  takes  us  "Down". 

I  now  mark  the  nodes  which  we  calculated  the  heuristics  for,  but  did  not  move  to,  as  orange,  and  the  node  which 
we  chose  as  cyan: 
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Alright,  now  let's  calculate  the  same  heuristics  for  the  nodes  around  the  cyan  node: 


Again,  we  choose  the  node  going  down  from  the  cyan  node,  as  all  the  options  have  the  same  f  value: 
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Let's  calculate  the  heuristics  for  the  only  neighbour  that  the  cyan  node  has: 


Alright,  since  we  will  follow  the  same  pattern  we  have  been  following: 
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Let's  move  there: 
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Finally,  we  can  see  that  we  have  a  winning  square  beside  us,  so  we  move  there,  and  we  are  done. 
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Chapter  8:  Dynamic  Programming 

Dynamic  programming  is  a  widely  used  concept  and  its  often  used  for  optimization.  It  refers  to  simplifying  a 
complicated  problem  by  breaking  it  down  into  simpler  sub-problems  in  a  recursive  manner  usually  a  bottom-up 
approach.  There  are  two  key  attributes  that  a  problem  must  have  in  order  for  dynamic  programming  to  be 
applicable  "Optimal  substructure"  and  "Overlapping  sub-problems".  To  achieve  its  optimization,  dynamic 
programming  uses  a  concept  called  memoization 

Section  8.1:  Edit  Distance 

The  problem  statement  is  like  if  we  are  given  two  string  strl  and  str2  then  how  many  minimum  number  of 
operations  can  be  performed  on  the  strl  that  it  gets  converted  to  str2. 

Implementation  in  Java 

public  class  EditDistance  { 

public  static  void  main (String [ ]  args)  { 

//  TODO  Auto-generated  method  stub 
String  strl  =  "march"; 

String  str2  =  "cart"; 

EditDistance  ed  =  new  EditDistance(); 

System. out . println(ed . getMinConversions(str1 ,  str2) ) ; 

} 

public  int  getMinConversions(String  strl,  String  str2){ 

int  dp[][]  =  new  int [ strl . length( )+1 ] [str2 . length( )+1 ] ; 
for(int  i=0 ; i<=st rl . length( )  i++){ 

for (int  j=0 ; j<=str2 . length ( ) ; j++) { 
if ( i==0) 

dp[i][j]  =  j; 
else  if(j==0) 

dp [ i ] [ j ]  =  i 

else  if (strl .charAt(i-1 )  ==  str2.charAt( j-1 )) 
dp [ i ] [ j ]  =  dp [ i  1  ]  [  j-1  ]  ; 

else{ 

dp[i][j]  =  1  +  Math .min (dp[ i -1 ][ j  ] ,  Math. min(dp[i] [j-1 ] ,  dp [ i-1  ]  [ j-1  ] ) )  ; 

} 

} 

} 

return  dp [strl  .  length ( ) ] [ st r2 . length ( ) ]  ; 

} 

} 

Output 

3 

Section  8.2:  Weighted  Job  Scheduling  Algorithm 

Weighted  Job  Scheduling  Algorithm  can  also  be  denoted  as  Weighted  Activity  Selection  Algorithm. 

The  problem  is,  given  certain  jobs  with  their  start  time  and  end  time,  and  a  profit  you  make  when  you  finish  the  job, 
what  is  the  maximum  profit  you  can  make  given  no  two  jobs  can  be  executed  in  parallel? 
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This  one  looks  like  Activity  Selection  using  Greedy  Algorithm,  but  there's  an  added  twist.  That  is,  instead  of 
maximizing  the  number  of  jobs  finished,  we  focus  on  making  the  maximum  profit.  The  number  of  jobs  performed 
doesn't  matter  here. 

Let's  look  at  an  example: 


Name 

. -+-■ 

1 

A 

-  +  - 

1 

B 

-  +  -  - 

1 

C 

-+-  - 

1 

D 

-  +  -  - 

1 

E 

-  +  - 

1 

F 

(Start  Time,  Finish 

Time) | 

(2,5) 

1 

(6,7) 

1 

(7,9) 

1 

(1,3) 

1 

(5,8) 

1 

(4,6) 

Profit 

1 

. -  +  -  - 

6 

1 

4 

1 

-  +  -  - 

2 

1 

-  +  -  - 

5 

1 

-  +  -  - 

11 

1 

5 

The  jobs  are  denoted  with  a  name,  their  start  and  finishing  time  and  profit.  After  a  few  iterations,  we  can  find  out  if 
we  perform  Job-A  and  Job-E,  we  can  get  the  maximum  profit  of  1 7.  Now  how  to  find  this  out  using  an  algorithm? 

The  first  thing  we  do  is  sort  the  jobs  by  their  finishing  time  in  non-decreasing  order.  Why  do  we  do  this?  It's  because 
if  we  select  a  job  that  takes  less  time  to  finish,  then  we  leave  the  most  amount  of  time  for  choosing  other  jobs.  We 
have: 


Name 

. -+-■ 

1 

D 

1 

A 

-  +  -  - 

1 

F 

-  +  - 

1 

B 

-  +  -  - 

1 

E 

-  +  - 

1 

C 

(Start  Time,  Finish 

Time) | 

(1,3) 

1 

(2,5) 

1 

(4,6) 

1 

(6,7) 

1 

(5,8) 

1 

(7,9) 

Profit 

1 

. -+-■ 

5 

1 

-  +  -  - 

6 

1 

-  +  -  - 

5 

1 

4 

1 

- 

11 

1 

2 

We'll  have  an  additional  temporary  array  Acc_Prof  of  size  n  (Here,  n  denotes  the  total  number  of  jobs).  This  will 
contain  the  maximum  accumulated  profit  of  performing  the  jobs.  Don't  get  it?  Wait  and  watch.  We'll  initialize  the 
values  of  the  array  with  the  profit  of  each  jobs.  That  means,  Acc_Prof[i]  will  at  first  hold  the  profit  of  performing  i-th 
job. 


+ - 

|  Acc_Prof 

+ - 


+ - + - + - + - + - + - + 


|  5  |  6  |  5  |  4  |  11  |  2  | 


+ - + - + - + - + - + - + 


Now  let's  denote  position  2  with  i,  and  position  1  will  be  denoted  with  j.  Our  strategy  will  be  to  iterate  j  from  1  to 
i-1  and  after  each  iteration,  we  will  increment  i  by  1 ,  until  i  becomes  n+1. 


j  i 


Name 

. -+-■ 

1 

D 

-  +  -  - 

1 

A 

-  +  -  - 

1 

F 

-+- 

1 

B 

-+-  - 

1 

E 

-  +  - 

1 

C 

(Start  Time,  Finish 

Time) | 

(1,3) 

1 

(2,5) 

1 

(4,6) 

1 

(6,7) 

1 

(5,8) 

1 

(7,9) 

Profit 

1 

5 

1 

6 

1 

5 

1 

4 

1 

11 

1 

2 

Acc_Prof 

1 

. -+-■ 

5 

1 

-  +  -  - 

6 

1 

-  +  -  - 

5 

1 

4 

1 

- 

11 

1 

2 

GoalKicker.com  -  Algorithms  Notes  for  Professionals 


45 


We  check  if  Job[i]  and  Job|J]  overlap,  that  is,  if  the  finish  time  of  Job[j]  is  greater  than  Job[i]'s  start  time,  then  these 
two  jobs  can't  be  done  together.  However,  if  they  don't  overlap,  we'll  check  if  Acc_Prof|J]  +  Profit[i]  >  Acc_Prof[i],  If 
this  is  the  case,  we  will  update  Acc_Prof[i]  =  Acc_Prof[j]  +  Profit[i],  That  is: 

if  Job[ j ] .finish_time  <=  Jobji  .start_time 
if  Acc_Prof[j]  +  Profitji]  >  Acc_Prof[i] 

Acc_Prof[i]  =  Acc_Prof[j]  +  Profitji] 
endif 
endif 

Here  Acc_Prof[j]  +  Profitji]  represents  the  accumulated  profit  of  doing  these  two  jobs  toegther.  Let's  check  it  for 
our  example: 

Here  Job|J]  overlaps  with  Jobji],  So  these  to  can't  be  done  together.  Since  our  j  is  equal  to  i-1,  we  increment  the 
value  of  i  to  i+1  that  is  3.  And  we  make  j  =  1. 


j  i 


Name 

- +  -  - 

1 

D 

1 

A 

-  +  -  - 

1 

F 

-  +  - 

1 

B 

1 

E 

-  +  - 

1 

C 

(Start  Time,  Finish 

Time) | 

(1,3) 

1 

(2,5) 

1 

(4,6) 

1 

(6,7) 

1 

(5,8) 

1 

(7,9) 

Profit 

1 

5 

1 

6 

1 

5 

1 

4 

1 

11 

1 

2 

Acc_Prof 

1 

- +  -  - 

5 

1 

-  +  -  - 

6 

1 

-  +  -  - 

5 

1 

4 

1 

-  +  -  - 

11 

1 

2 

Now  Jobjj]  and  Jobji]  don't  overlap.  The  total  amount  of  profit  we  can  make  by  picking  these  two  jobs  is:  Acc_Profjj] 
+  Profitji]  =  5  +  5  =  10  which  is  greater  than  Acc_Prof[i],  So  we  update  Acc_Prof[i]  =  10.  We  also  increment  j  by  1 . 
We  get, 


j  i 


Name 

- +  -  - 

1 

D 

-  +  -  - 

1 

A 

-  +  -  - 

1 

F 

-  +  - 

1 

B 

-  +  -  - 

1 

E 

-  +  - 

1 

C 

(Start  Time,  Finish 

Time) | 

(1,3) 

1 

(2,5) 

1 

(4,6) 

1 

(6,7) 

1 

(5,8) 

1 

(7,9) 

Profit 

1 

5 

1 

6 

1 

5 

1 

4 

1 

11 

1 

2 

Acc_Prof 

1 

. -  +  -  - 

5 

1 

6 

1 

-  +  -  - 

10 

1 

4 

1 

-  +  -  - 

11 

1 

2 

Here,  Jobjj]  overlaps  with  Jobji]  and  j  is  also  equal  to  i-1.  So  we  increment  i  by  1,  and  make  j  =  1.  We  get, 


j  i 


Name 

. -+-■ 

1 

D 

-  +  -  - 

1 

A 

-  +  -  - 

1 

F 

-+- 

1 

B 

-+-  - 

1 

E 

-  +  - 

1 

C 

(Start  Time,  Finish 

Time) | 

(1,3) 

1 

(2,5) 

1 

(4,6) 

1 

(6,7) 

1 

(5,8) 

1 

(7,9) 

Profit 

1 

- +  -  - 

5 

1 

-  +  -  - 

6 

1 

-  +  -  - 

5 

1 

4 

1 

11 

1 

2 

Acc_Prof  |  5  |  6  |  10  |  4  |  11  |  2 
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Now,  Job[j]  and  Job[i]  don't  overlap,  we  get  the  accumulated  profit  5  +  4  =  9,  which  is  greater  than  Acc_Prof[i],  We 
update  Acc_Prof[i]  =  9  and  increment  j  by  1 . 


j  i 


Name 

. -  +  -  - 

1 

D 

-  +  -  - 

1 

A 

-  +  -  - 

1 

F 

-  +  - 

1 

B 

-  +  -  - 

1 

E 

-  +  - 

1 

C 

(Start  Time,  Finish 

Time) | 

(1,3) 

1 

(2,5) 

1 

(4,6) 

1 

(6,7) 

1 

(5,8) 

1 

(7,9) 

Profit 

1 

5 

1 

6 

1 

5 

1 

4 

1 

11 

1 

2 

Acc_Prof 

1 

- +  -  - 

5 

1 

-  +  -  - 

6 

1 

-  +  -  - 

10 

1 

9 

1 

-  +  -  - 

11 

1 

2 

Again  Job[j]  and  Job[i]  don't  overlap.  The  accumulated  profit  is:  6  +  4  =  10,  which  is  greater  than  Acc_Prof[i],  We 
again  update  Acc_Prof[i]  =  10.  We  increment  j  by  1 .  We  get: 


j  i 


Name 

. -  +  -  - 

1 

D 

-  +  -  - 

1 

A 

-  +  -  - 

1 

F 

-  +  - 

1 

B 

-+-  - 

1 

E 

-  +  - 

1 

C 

(Start  Time,  Finish 

Time) | 

(1,3) 

1 

(2,5) 

1 

(4,6) 

1 

(6,7) 

1 

(5,8) 

1 

(7,9) 

Profit 

1 

5 

1 

6 

1 

5 

1 

4 

1 

11 

1 

2 

Acc_Prof 

1 

- +  -  - 

5 

1 

-  +  -  - 

6 

1 

-  +  -  - 

10 

1 

10 

1 

11 

1 

2 

If  we  continue  this  process,  after  iterating  through  the  whole  table  using  i,  our  table  will  finally  look  like: 


Name 

. -+-■ 

1 

D 

-  +  -  - 

1 

A 

-  +  -  - 

1 

F 

-  +  - 

1 

B 

-+-  - 

1 

E 

-  +  - 

1 

C 

(Start  Time,  Finish 

Time) | 

(1,3) 

1 

(2,5) 

1 

(4,6) 

1 

(6,7) 

1 

(5,8) 

1 

(7,9) 

Profit 

1 

5 

1 

6 

1 

5 

1 

4 

1 

11 

1 

2 

Acc_Prof 

1 

. -+-■ 

5 

1 

-  +  -  - 

6 

1 

-  +  -  - 

10 

1 

14 

1 

- 

17 

1 

8 

*  A  few  steps  have  been  skipped  to  make  the  document  shorter. 

If  we  iterate  through  the  array  Acc_Prof,  we  can  find  out  the  maximum  profit  to  be  17!  The  pseudo-code: 
Procedure  WeightedJobScheduling(Job) 

sort  Job  according  to  finish  time  in  non -decreasing  order 
for  i  ->  2  to  n 

for  j  ->  1  to  i-1 

if  Job[ j ] .finish_time  <=  Job[i] .start_time 
if  Acc_Prof[j]  +  Profit[i]  >  Acc_Prof[i] 

Acc_Prof[i]  =  Acc_Prof[j]  +  Profit[i] 
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endif 

endif 

endfor 

endfor 

maxProfit  =  0 
for  i  ->  1  to  n 

if  maxProfit  <  Acc_Prof[i 
maxProfit  =  Acc_Prof[i 
return  maxProfit 

The  complexity  of  populating  the  Acc_Prof  array  is  0(n2).  The  array  traversal  takes  O(n).  So  the  total  complexity  of 
this  algorithm  is  0(n2). 

Now,  If  we  want  to  find  out  which  jobs  were  performed  to  get  the  maximum  profit,  we  need  to  traverse  the  array  in 
reverse  order  and  if  the  Acc_Prof  matches  the  maxProfit,  we  will  push  the  name  of  the  job  in  a  stack  and  subtract 
Profit  of  that  job  from  maxProfit.  We  will  do  this  until  our  maxProfit  >  0  or  we  reach  the  beginning  point  of  the 
Acc_Prof  array.  The  pseudo-code  will  look  like: 

Procedure  FindingPerformedJobs( Job,  Acc_Prof,  maxProfit): 

S  =  stack() 

for  i  ->  n  down  to  0  and  maxProfit  >  0 
if  maxProfit  is  equal  to  Acc_Prof[i 
S.push(Job[i] .name 

maxProfit  =  maxProfit  -  Job[i  .profit 
endif 
endfor 

The  complexity  of  this  procedure  is:  O(n). 

One  thing  to  remember,  if  there  are  multiple  job  schedules  that  can  give  us  maximum  profit,  we  can  only  find  one 
job  schedule  via  this  procedure. 

Section  8.3:  Longest  Common  Subsequence 

If  we  are  given  with  the  two  strings  we  have  to  find  the  longest  common  sub-sequence  present  in  both  of  them. 

Example 

LCS  for  input  Sequences  “ABCDGH”  and  “AEDFHR”  is  “ADH”  of  length  3. 

LCS  for  input  Sequences  “AGGTAB”  and  “GXTXAYB”  is  “GTAB”  of  length  4. 

Implementation  in  Java 

public  class  LCS  { 

public  static  void  main(String[ ]  args)  { 

//  TODO  Auto-generated  method  stub 
String  strl  =  "AGGTAB"; 

String  str2  =  "GXTXAYB"; 

LCS  obj  =  new  LCS(); 

System. out . println(obj . Ics(str1 ,  str2,  strl . length( ) ,  str2 . length( ) ) ) ; 

System. out . println( obj . Ics2( strl ,  str2) ) ; 

} 

//Recursive  function 

public  int  lcs(String  strl,  String  str2,  int  m,  int  n){ 
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if(m==0  | |  n==0) 

return  0; 

if (strl .charAt(m-1 )  ==  str2.charAt(n  1)) 
return  1  +  lcs(str1,  str2,  m-1,  n-1); 

else 

return  Math  .max(lcs(str1 ,  str2,  m-1,  n) ,  lcs(str1,  str2,  m,  n-1)); 


//Iterative  function 

public  int  lcs2(String  strl,  String  str2){ 

int  lcs[][]  =  new  int [strl . length( )+1 ][ str2 . length(  )+1  ]  ; 

for(int  i=0;i<=str1.1ength(); i++) { 

for (int  j=0 ; j<=st r2 . length ( ) ; j++) { 
if ( i==0  | |  j==  0){ 
lcs [i] [ j ]  =  0; 

} 

else  if (strl .charAt(i  1)  ==  str2 . charAt ( j -1 ) ) { 
lcs [i] [ j ]  =  1  +  lcs[i-1 ]  [  j-1  ]  ; 

}else{ 

lcs [ i ] [ j ]  =  Math.max(lcs[i -1 ] [ j ] ,  lcs[i] [ j-1 ] ) ; 

} 

} 

} 

return  lcs[ strl . length( ) ] [str2 . length( ) ] ; 

} 


} 

Output 

4 

Section  8.4:  Fibonacci  Number 

Bottom  up  approach  for  printing  the  nth  Fibonacci  number  using  Dynamic  Programming. 

Recursive  Tree 


f ib( 5) 

/  \ 

fib(4)  fib(3) 

/  \  /  \ 
f ib(3)  f ib ( 2 )  fib(2)  fib(1) 

/  \  /  \  /  \ 
fib(2)  fib(1)  f ib ( 1 )  fib(0)  fib(1)  fib(0) 

/  \ 

fib(1)  fib(0) 

Overlapping  Sub-problems 

Here  fib(0),fib(1 )  and  fib(3)  are  the  overlapping  sub-problems.fib(O)  is  getting  repeated  3  times,  fib(1 )  is  getting 
repeated  5  times  and  fib(3)  is  getting  repeated  2  times. 

Implementation 

public  int  fib(int  n){ 

int  f[]  =  new  int[n+1]; 
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f [0 ] =0 ; f [ 1 ] =1  ; 
for(int  i=2  i<=n  i++){ 
f[i]=f[i-1]+f[i  2]; 

} 

return  f [n]  ; 


Time  Complexity 

O(n) 

Section  8.5:  Longest  Common  Substring 

Given  2  string  strl  and  str2  we  have  to  find  the  length  of  the  longest  common  substring  between  them. 

Examples 

Input :  X  =  "abcdxyz",  y  =  "xyzabcd"  Output :  4 

The  longest  common  substring  is  "abed"  and  is  of  length  4. 

Input :  X  =  "zxabedezy",  y  =  "yzabcdezx"  Output :  6 

The  longest  common  substring  is  "abedez"  and  is  of  length  6. 

Implementation  in  Java 

public  int  getLongestCommonSubstring(String  strl .String  str2){ 

int  arr[][]  =  new  int [str2 . length( )+1 ][ strl . length( )+1 ] ; 
int  max  =  Integer . MIN_VALUE ; 
for(int  i=1  i<=str2 . length( )  i++){ 

for(int  j=1  ;j<  =  str1 .length();j++){ 

if (strl .charAt( j-1 )  ==  str2 .charAt(i  1)){ 
arr [i] [ j ]  =  arr [ i -1 ] [ j-1 ]+1 ; 
if(arr[i  [ j ]>max) 
max  =  arr[i] [ j ] ; 

} 

else 

arr [i] [ j ]  =  0; 

} 

} 

return  max; 


Time  Complexity 

0(m*n) 
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Chapter  9:  Kruskal's  Algorithm 

Section  9.1:  Optimal,  disjoint-set  based  implementation 

We  can  do  two  things  to  improve  the  simple  and  sub-optimal  disjoint-set  subalgorithms: 

1 .  Path  compression  heuristic:  f  indSet  does  not  need  to  ever  handle  a  tree  with  height  bigger  than  2.  If  it 
ends  up  iterating  such  a  tree,  it  can  link  the  lower  nodes  directly  to  the  root,  optimizing  future  traversals; 

subalgo  findSet(v  a  node): 
if  v. parent  !=  v 

v. parent  =  findSet(v. parent) 
return  v. parent 

2.  Height-based  merging  heuristic:  for  each  node,  store  the  height  of  its  subtree.  When  merging,  make  the 
taller  tree  the  parent  of  the  smaller  one,  thus  not  increasing  anyone's  height. 


subalgo  unionSet(u,  v  nodes)  : 
vRoot  =  findSet(v) 
uRoot  =  findSet(u) 

if  vRoot  ==  uRoot : 

return 

if  vRoot. height  <  uRoot . height : 

vRoot. parent  =  uRoot 
else  if  vRoot. height  >  uRoot . height : 
uRoot. parent  =  vRoot 

else 

uRoot. parent  =  vRoot 

uRoot. height  =  uRoot. height  +  1 


This  leads  to  0(  alpha  (n) )  time  for  each  operation,  where  alpha  is  the  inverse  of  the  fast-growing  Ackermann 
function,  thus  it  is  very  slow  growing,  and  can  be  considered  0(1)  for  practical  purposes. 

This  makes  the  entire  Kruskal's  algorithm  0(m  log  m  +  m)  =  0(m  log  m),  because  of  the  initial  sorting. 

Note 

Path  compression  may  reduce  the  height  of  the  tree,  hence  comparing  heights  of  the  trees  during  union  operation 
might  not  be  a  trivial  task.  Hence  to  avoid  the  complexity  of  storing  and  calculating  the  height  of  the  trees  the 
resulting  parent  can  be  picked  randomly: 

subalgo  unionSet(u,  v  nodes): 
vRoot  =  findSet(v) 
uRoot  =  findSet(u) 

if  vRoot  ==  uRoot : 

return 

if  random()  X  2  ==  0: 
vRoot. parent  =  uRoot 

else 

uRoot. parent  =  vRoot 

In  practice  this  randomised  algorithm  together  with  path  compression  for  f  indSet  operation  will  result  in 
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comparable  performance,  yet  much  simpler  to  implement. 

Section  9.2:  Simple,  more  detailed  implementation 

In  order  to  efficiently  handle  cycle  detection,  we  consider  each  node  as  part  of  a  tree.  When  adding  an  edge,  we 
check  if  its  two  component  nodes  are  part  of  distinct  trees.  Initially,  each  node  makes  up  a  one-node  tree. 

algorithm  kruskalMST' (G :  a  graph) 
sort  G's  edges  by  their  value 

MST  =  a  forest  of  trees,  initially  each  tree  is  a  node  in  the  graph 
for  each  edge  e  in  G : 

if  the  root  of  the  tree  that  e. first  belongs  to  is  not  the  same 
as  the  root  of  the  tree  that  e. second  belongs  to 

connect  one  of  the  roots  to  the  other,  thus  merging  two  trees 

return  MST,  which  now  a  single  tree  forest 

Section  9.3:  Simple,  disjoint-set  based  implementation 

The  above  forest  methodology  is  actually  a  disjoint-set  data  structure,  which  involves  three  main  operations: 

subalgo  makeSet(v:  a  node): 

v. parent  =  v  <-  make  a  new  tree  rooted  at  v 


subalgo  findSet(v:  a  node): 
if  v. parent  ==  v: 

return  v 

return  findSet(v. parent) 

subalgo  unionSet(v,  u  nodes) : 
vRoot  =  findSet(v) 
uRoot  =  findSet(u) 

uRoot. parent  =  vRoot 

algorithm  kruskalMST' ' (G :  a  graph): 
sort  G's  edges  by  their  value 
for  each  node  n  in  G: 
makeSet(n) 

for  each  edge  e  in  G: 

if  findSet(e. first)  !=  findSet(e. second) : 
unionSet(e. first,  e. second) 

This  naive  implementation  leads  to  0(n  log  n)  time  for  managing  the  disjoint-set  data  structure,  leading  to  0(m*n 
log  n)  time  for  the  entire  Kruskal's  algorithm. 

Section  9.4:  Simple,  high  level  implementation 

Sort  the  edges  by  value  and  add  each  one  to  the  MST  in  sorted  order,  if  it  doesn't  create  a  cycle. 

algorithm  kruskalMST(G :  a  graph) 
sort  G's  edges  by  their  value 
MST  =  an  empty  graph 
for  each  edge  e  in  G: 

if  adding  e  to  MST  does  not  create  a  cycle: 
add  e  to  MST 
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return  MST 
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Chapter  10:  Greedy  Algorithms 

Section  10.1:  Huffman  Coding 

Huffman  code  is  a  particular  type  of  optimal  prefix  code  that  is  commonly  used  for  lossless  data  compression.  It 
compresses  data  very  effectively  saving  from  20%  to  90%  memory,  depending  on  the  characteristics  of  the  data 
being  compressed.  We  consider  the  data  to  be  a  sequence  of  characters.  Huffman's  greedy  algorithm  uses  a  table 
giving  how  often  each  character  occurs  (i.e.,  its  frequency)  to  build  up  an  optimal  way  of  representing  each 
character  as  a  binary  string.  Huffman  code  was  proposed  by  David  A.  Huffman  in  1 951 . 

Suppose  we  have  a  1 00,000-character  data  file  that  we  wish  to  store  compactly.  We  assume  that  there  are  only  6 
different  characters  in  that  file.  The  frequency  of  the  characters  are  given  by: 

+ - + - + - + - + - + - + - + 

|  Character  |a|b|c|d|e|f| 

+ - + - + - + - + - + - + - + 

| Frequency  (in  thousands)!  45  |  13  |  12  |  16  |  9  |  5  | 

+ - + - + - + - + - + - + - + 


We  have  many  options  for  how  to  represent  such  a  file  of  information.  Here,  we  consider  the  problem  of  designing 
a  Binary  Character  Code  in  which  each  character  is  represented  by  a  unique  binary  string,  which  we  call  a  codeword. 


Fixed-length  Codeword 


The  constructed  tree  will  provide  us  with: 

+ - + - + - + - + - + - + - + 

|  Character  |a|b|c|d|e|f| 

+ - + - + - + - + - + - + - + 

|  Fixed-length  Codeword  |  000  |  001  |  010  |  011  |  100  |  101  | 

+ - + - + - + - + - + - + - + 

| Variable-length  Codeword!  0  I  101  I  100  I  HI  I  1101 |  1100 | 

+ - + - + - + - + - + - + - + 

If  we  use  a  fixed-length  code,  we  need  three  bits  to  represent  6  characters.  This  method  requires  300,000  bits  to 
code  the  entire  file.  Now  the  question  is,  can  we  do  better? 
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A  variable-length  code  can  do  considerably  better  than  a  fixed-length  code,  by  giving  frequent  characters  short 
codewords  and  infrequent  characters  long  codewords.  This  code  requires:  (45  XI  +13X3+12X3+16X3+9X4+ 
5  X  4)  X  1 000  =  224000  bits  to  represent  the  file,  which  saves  approximately  25%  of  memory. 

One  thing  to  remember,  we  consider  here  only  codes  in  which  no  codeword  is  also  a  prefix  of  some  other 
codeword.  These  are  called  prefix  codes.  For  variable-length  coding,  we  code  the  3-character  file  abc  as  0.1 01 .1 00  = 
01 01 1 00,  where  denotes  the  concatenation. 

Prefix  codes  are  desirable  because  they  simplify  decoding.  Since  no  codeword  is  a  prefix  of  any  other,  the 
codeword  that  begins  an  encoded  file  is  unambiguous.  We  can  simply  identify  the  initial  codeword,  translate  it  back 
to  the  original  character,  and  repeat  the  decoding  process  on  the  remainder  of  the  encoded  file.  For  example, 

001 011101  parses  uniquely  as  0.0.1 01 .1101,  which  decodes  to  aabe.  In  short,  all  the  combinations  of  binary 
representations  are  unique.  Say  for  example,  if  one  letter  is  denoted  by  1 1 0,  no  other  letter  will  be  denoted  by  1 1 01 
or  1 1 00.  This  is  because  you  might  face  confusion  on  whether  to  select  1 1 0  or  to  continue  on  concatenating  the 
next  bit  and  select  that  one. 

Compression  Technique: 

The  technique  works  by  creating  a  binary  tree  of  nodes.  These  can  stored  in  a  regular  array,  the  size  of  which 
depends  on  the  number  of  symbols,  n.  A  node  can  either  be  a  leaf  node  or  an  internal  node.  Initially  all  nodes  are 
leaf  nodes,  which  contain  the  symbol  itself,  its  frequency  and  optionally,  a  link  to  its  child  nodes.  As  a  convention, 
bit  'O'  represents  left  child  and  bit  '1 '  represents  right  child.  Priority  queue  is  used  to  store  the  nodes,  which  provides 
the  node  with  lowest  frequency  when  popped.  The  process  is  described  below: 

1 .  Create  a  leaf  node  for  each  symbol  and  add  it  to  the  priority  queue. 

2.  While  there  is  more  than  one  node  in  the  queue: 

1 .  Remove  the  two  nodes  of  highest  priority  from  the  queue. 

2.  Create  a  new  internal  node  with  these  two  nodes  as  children  and  with  frequency  equal  to  the  sum  of 
the  two  nodes'  frequency. 

3.  Add  the  new  node  to  the  queue. 

3.  The  remaining  node  is  the  root  node  and  the  Fluffman  tree  is  complete. 

For  our  example: 
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The  pseudo-code  looks  like: 

Procedure  Huffman(C):  //  C  is  the  set  of  n  characters  and  related  information 

n  =  C.size 

Q  =  priority_queue( ) 
for  i  =  1  to  n 

n  =  node(C[i]) 

Q.push(n) 

end  for 

while  Q.size()  is  not  equal  to  1 
Z  =  new  node() 

Z.left  =  x  =  Q.pop 
Z . right  =  y  =  Q . pop 

Z. frequency  =  x. frequency  +  y. frequency 
Q . push(Z) 

end  while 
Return  Q 

Although  linear-time  given  sorted  input,  in  general  cases  of  arbitrary  input,  using  this  algorithm  requires  pre¬ 
sorting.  Thus,  since  sorting  takes  O(nlogn)  time  in  general  cases,  both  methods  have  same  complexity. 

Since  n  here  is  the  number  of  symbols  in  the  alphabet,  which  is  typically  very  small  number  (compared  to  the 
length  of  the  message  to  be  encoded),  time  complexity  is  not  very  important  in  the  choice  of  this  algorithm. 

Decompression  Technique: 

The  process  of  decompression  is  simply  a  matter  of  translating  the  stream  of  prefix  codes  to  individual  byte  value, 
usually  by  traversing  the  Huffman  tree  node  by  node  as  each  bit  is  read  from  the  input  stream.  Reaching  a  leaf 
node  necessarily  terminates  the  search  for  that  particular  byte  value.  The  leaf  value  represents  the  desired 
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character.  Usually  the  Huffman  Tree  is  constructed  using  statistically  adjusted  data  on  each  compression  cycle,  thus 
the  reconstruction  is  fairly  simple.  Otherwise,  the  information  to  reconstruct  the  tree  must  be  sent  separately.  The 
pseudo-code: 

Procedure  Huf fmanDecompression( root ,  S) :  //  root  represents  the  root  of  Huffman  Tree 

n  :=  S. length  //  S  refers  to  bit-stream  to  be  decompressed 

for  i  :=  1  to  n 
current  =  root 

while  current. left  !=  NULL  and  current . right  !=  NULL 
if  S[i  is  equal  to  '0' 

current  :=  current. left 

else 

current  :=  current . right 
endif 

i  :=  i+1 

endwhile 

print  current .symbol 
endfor 

Greedy  Explanation: 

Huffman  coding  looks  at  the  occurrence  of  each  character  and  stores  it  as  a  binary  string  in  an  optimal  way.  The 
idea  is  to  assign  variable-length  codes  to  input  input  characters,  length  of  the  assigned  codes  are  based  on  the 
frequencies  of  corresponding  characters.  We  create  a  binary  tree  and  operate  on  it  in  bottom-up  manner  so  that 
the  least  two  frequent  characters  are  as  far  as  possible  from  the  root.  In  this  way,  the  most  frequent  character  gets 
the  smallest  code  and  the  least  frequent  character  gets  the  largest  code. 

References: 

•  Introduction  to  Algorithms  -  Charles  E.  Leiserson,  Clifford  Stein,  Ronald  Rivest,  and  Thomas  H.  Cormen 

•  Huffman  Coding  -  Wikipedia 

•  Discrete  Mathematics  and  Its  Applications  -  Kenneth  H.  Rosen 

Section  10.2:  Activity  Selection  Problem 

The  Problem 

You  have  a  set  of  things  to  do  (activities).  Each  activity  has  a  start  time  and  a  end  time.  You  aren't  allowed  to 
perform  more  than  one  activity  at  a  time.  Your  task  is  to  find  a  way  to  perform  the  maximum  number  of  activities. 

For  example,  suppose  you  have  a  selection  of  classes  to  choose  from. 

Activity  No.  start  time  end  time 

1  1 0.20  A.M  11.00AM 

2  10.30  A.M  11.30AM 

3  11. 00  A.M  12.00AM 

4  10.00  A.M  11.30AM 

5  9.00  A.M  11.00AM 

Remember,  you  can't  take  two  classes  at  the  same  time.  That  means  you  can't  take  class  1  and  2  because  they 
share  a  common  time  1 0.30  A.M  to  1 1 .00  A.M.  However,  you  can  take  class  1  and  3  because  they  don't  share  a 
common  time.  So  your  task  is  to  take  maximum  number  of  classes  as  possible  without  any  overlap.  How  can  you 
do  that? 

Analysis 

Lets  think  for  the  solution  by  greedy  approach. First  of  all  we  randomly  chose  some  approach  and  check  that  will 
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work  or  not. 


•  sort  the  activity  by  start  time  that  means  which  activity  start  first  we  will  take  them  first,  then  take  first  to 
last  from  sorted  list  and  check  it  will  intersect  from  previous  taken  activity  or  not.  If  the  current  activity  is  not 
intersect  with  the  previously  taken  activity,  we  will  perform  the  activity  otherwise  we  will  not  perform,  this 
approach  will  work  for  some  cases  like 


Activity  No.  start  time  end  time 

1  11.00A.M  1.30P.M 

2  11.30A.M  1 2.00P.M 

3  1.30  P.M  2.00P.M 

4  1 0.00  A.M  11.00AM 


the  sorting  order  will  be  4-->1  ->2-->3  .The  activity  4~>  1  -->  3  will  be  performed  and  the  activity  2  will  be  skipped, 
the  maximum  3  activity  will  be  performed.  It  works  for  this  type  of  cases,  but  it  will  fail  for  some  cases.  Lets  apply 
this  approach  for  the  case 


Activity  No.  start  time  end  time 

1  11. 00  A.M  1.30P.M 

2  11. 30  A.M  1 2.00P.M 

3  1.30  P.M  2.00P.M 

4  10.00  A.M  3.00P.M 


The  sort  order  will  be  4~>1  ~>2~>3  and  only  activity  4  will  be  performed  but  the  answer  can  be  activity  1  -->3  or  2- 
->3  will  be  performed.  So  our  approach  will  not  work  for  the  above  case.  Let's  try  another  approach 


•  Sort  the  activity  by  time  duration  that  means  perform  the  shortest  activity  first,  that  can  solve  the  previous 
problem  .  Although  the  problem  is  not  completely  solved.  There  still  some  cases  that  can  fail  the  solution, 
apply  this  approach  on  the  case  bellow. 

Activity  No.  start  time  end  time 

1  6.00  A.M  11.40A.M 

2  11. 30  A.M  1 2.00P.M 

3  11.40  P.M  2.00P.M 


if  we  sort  the  activity  by  time  duration  the  sort  order  will  be  2->  3  — >1  .  and  if  we  perform  activity  No.  2  first  then 
no  other  activity  can  be  performed.  But  the  answer  will  be  perform  activity  1  then  perform  3  .  So  we  can  perform 
maximum  2  activity.So  this  can  not  be  a  solution  of  this  problem.  We  should  try  a  different  approach. 


The  solution 

•  Sort  the  Activity  by  ending  time  that  means  the  activity  finishes  first  that  come  first,  the  algorithm  is  given 
below 

1 .  Sort  the  activities  by  its  ending  times. 

2.  If  the  activity  to  be  performed  do  not  share  a  common  time  with  the  activities  that  previously 
performed,  perform  the  activity. 

Lets  analyse  the  first  example 

Activity  No.  start  time  end  time 

1  10.20  A.M  11.00AM 

2  10.30  A.M  11.30AM 

3  11. 00  A.M  12.00AM 
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4 

5 


1 0.00  A.M  11.30AM 
9.00  A. M  11.00AM 


sort  the  activity  by  its  ending  times  ,  So  sort  order  will  be  1->5-->2->4-->3..  the  answer  is  1  ~>3  these  two  activities 
will  be  performed,  ans  that's  the  answer,  here  is  the  sudo  code. 

1.  sort:  activities 

2.  perform  first  activity  from  the  sorted  list  of  activities. 

3.  Set :  Current_activity  :=  first  activity 

4.  set:  end_time  :=  end_time  of  Current  activity 

5.  go  to  next  activity  if  exist,  if  not  exist  terminate  . 

6.  if  start_time  of  current  activity  <=  end _time  :  perform  the  activity  and  go  to  4 

7.  else:  got  to  5. 


see  here  for  coding  help  http://www.geeksforgeeks.org/greedy-algorithms-set-1-activity-selection-problem/ 

Section  10.3:  Change-making  problem 

Given  a  money  system,  is  it  possible  to  give  an  amount  of  coins  and  how  to  find  a  minimal  set  of  coins 
corresponding  to  this  amount. 

Canonical  money  systems.  For  some  money  system,  like  the  ones  we  use  in  the  real  life,  the  "intuitive"  solution 
works  perfectly.  For  example,  if  the  different  euro  coins  and  bills  (excluding  cents)  are  1€,  2€,  5€,  1 0€,  giving  the 
highest  coin  or  bill  until  we  reach  the  amount  and  repeating  this  procedure  will  lead  to  the  minimal  set  of  coins. 

We  can  do  that  recursively  with  OCaml : 

(*  assuming  the  money  system  is  sorted  in  decreasing  order  *) 
let  change_make  money_system  amount  = 
let  rec  loop  given  amount  = 
if  amount  =  0  then  given 

else 

(*  we  find  the  first  value  smaller  or  equal  to  the  remaining  amount  *) 
let  coin  =  List. find  ((>=)  amount)  money_system  in 
loop  (coin  :given)  (amount  -  coin) 
in  loop  [ ]  amount 

These  systems  are  made  so  that  change-making  is  easy.  The  problem  gets  harder  when  it  comes  to  arbitrary  money 
system. 

General  case.  Flow  to  give  99€  with  coins  of  1 0€,  7€  and  5€?  Flere,  giving  coins  of  1 0€  until  we  are  left  with  9€  leads 
obviously  to  no  solution.  Worse  than  that  a  solution  may  not  exist.  This  problem  is  in  fact  np-hard,  but  acceptable 
solutions  mixing  greediness  and  memoization  exist.  The  idea  is  to  explore  all  the  possibilies  and  pick  the  one  with 
the  minimal  number  of  coins. 

To  give  an  amount  X  >  0,  we  choose  a  piece  P  in  the  money  system,  and  then  solve  the  sub-problem  corresponding 
to  X-P.  We  try  this  for  all  the  pieces  of  the  system.  The  solution,  if  it  exists,  is  then  the  smallest  path  that  led  to  0. 

Flere  an  OCaml  recursive  function  corresponding  to  this  method.  It  returns  None,  if  no  solution  exists. 

(*  option  utilities  *) 
let  optmin  x  y  = 
match  x,y  with 
|  None, a  |  a, None  ->  a 


GoalKicker.com  -  Algorithms  Notes  for  Professionals 


59 


|  Some  x,  Some  y->  Some  (min  x  y) 


let  optsucc  =  function 
|  Some  x  ->  Some  (x+1 ) 

|  None  ->  None 

(*  Change-making  problem*) 
let  change_make  money_system  amount  = 
let  rec  loop  n  = 

let  onepiece  acc  piece  = 
match  n  piece  with 

|  0  ->  (*problem  solved  with  one  coin*) 

Some  1 

|  x  ->  if  x  <  0  then 

(*we  don't  reach  0,  we  discard  this  solution*) 

None 

else 

(*we  search  the  smallest  path  different  to  None  with  the  remaining  pieces*) 
optmin  (optsucc  (loop  x))  acc 
in 

(*we  call  onepiece  forall  the  pieces*) 

List . fold_lef t  onepiece  None  money_system 
in  loop  amount 

Note:  We  can  remark  that  this  procedure  may  compute  several  times  the  change  set  for  the  same  value.  In 
practice,  using  memoization  to  avoid  these  repetitions  leads  to  faster  (way  faster)  results. 
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Chapter  11:  Applications  of  Greedy 
technique 

Section  11.1:  Offline  Caching 

The  caching  problem  arises  from  the  limitation  of  finite  space.  Lets  assume  our  cache  C  has  k  pages.  Now  we  want 
to  process  a  sequence  of  m  item  requests  which  must  have  been  placed  in  the  cache  before  they  are  processed. Of 
course  if  m<=k  then  we  just  put  all  elements  in  the  cache  and  it  will  work,  but  usually  is  m>>k. 

We  say  a  request  is  a  cache  hit,  when  the  item  is  already  in  cache,  otherwise  its  called  a  cache  miss.  In  that  case 
we  must  bring  the  requested  item  into  cache  and  evict  another,  assuming  the  cache  is  full.  The  Goal  is  a  eviction 
schedule  that  minimizes  the  number  of  evictions. 

There  are  numerous  greedy  strategies  for  this  problem,  lets  look  at  some: 

1 .  First  in,  first  out  (FIFO):  The  oldest  page  gets  evicted 

2.  Last  in,  first  out  (LIFO):  The  newest  page  gets  evicted 

3.  Last  recent  out  (LRU):  Evict  page  whose  most  recent  access  was  earliest 

4.  Least  frequently  requested(LFU):  Evict  page  that  was  least  frequently  requested 

5.  Longest  forward  distance  (LFD):  Evict  page  in  the  cache  that  is  not  requested  until  farthest  in  the  future. 

Attention:  For  the  following  examples  we  evict  the  page  with  the  smallest  index,  if  more  than  one  page  could  be 
evicted. 

Example  (FIFO) 

Let  the  cache  size  be  k=3  the  initial  cache  a,  b,  c  and  the  request  a,a,d,e,b,b,a,c,f,d,e,a,f,b,e,c: 

Request  aadebbacfdeafbec 

cache  1  aaddddaaadddfffc 
cache  2  bbbeeeeccceeebbb 
cache  3  ccccbbbbf  f  f  aaaee 
cache  miss  xxx  xxxxxxxxxx 

Thirteen  cache  misses  by  sixteen  requests  does  not  sound  very  optimal,  lets  try  the  same  example  with  another 
strategy: 

Example  (LFD) 

Let  the  cache  size  be  k=3  the  initial  cache  a,  b,  c  and  the  request  a,a,d,e,b,b,a,c,f,d,e,a,f,b,e,c: 

Request  aadebbacfdeafbec 

cache  1  aadeeeeeeeeeeeec 
cache  2  bbbbbbaaaaaaffff 
cache  3  ccccccccfddddbbb 
cache  miss  xx  xxx  xxx 

Eight  cache  misses  is  a  lot  better. 

Selftest:  Do  the  example  for  LIFO,  LFU,  RFU  and  look  what  happend. 

The  following  example  programm  (written  in  C++)  consists  of  two  parts: 

The  skeleton  is  a  application,  which  solves  the  problem  dependent  on  the  chosen  greedy  strategy: 
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#include  <iostream> 
#include  <memory> 


using  namespace  std  ; 


const  int  cacheSize  =  3; 

const  int  requestLength  =  16; 


const  char  request!  ] 
char  cache! ] 


\  u  |  u  |  , 

{ '  a ' ,  '  b ' ,  '  c '  }  ; 


//  for  reset 

char  originalCache[ ]  =  { ' a  1 , ' b ' , ' c ' } ; 


class  Strategy  { 
public : 

Strategy(std : istring  name)  :  strategyName(name)  {} 
virtual  ~Strategy()  =  default; 

//  calculate  which  cache  place  should  be  used 

virtual  int  apply(int  requestlndex)  =  0; 

//  updates  information  the  strategy  needs 

virtual  void  update(int  cachePlace,  int  requestlndex,  bool  cacheMiss)  =  0; 
const  std::string  strategyName 

}  ; 

bool  updateCache( int  requestlndex,  Strategy*  strategy) 

{ 

//  calculate  where  to  put  request 

int  cachePlace  =  strategy->apply( requestlndex) ; 

//  proof  whether  its  a  cache  hit  or  a  cache  miss 

bool  isMiss  =  request [ requestlndex]  !=  cache[cachePlace] ; 

//  update  strategy  (for  example  recount  distances) 
strategy->update(cachePlace,  requestlndex,  isMiss); 

//  write  to  cache 

cache[cachePlace]  =  request [ requestlndex] ; 
return  isMiss; 


} 


int  main() 

{ 


Strategy*  selectedStrategy [ ]  =  {  new  FIFO,  new  LIFO,  new  LRU,  new  LFU,  new  LFD  }; 

for  (int  strat=0  strat  <  5  ++strat) 

{ 

//  reset  cache 

for  (int  i=0;  i  <  cacheSize  ++i)  cache[i]  =  originalCache[ i]  ; 

cout  <<"\nStrategy :  "  <<  selectedStrategy[strat]->strategyName  <<  endl; 

cout  <<  "\nCache  initial:  ("; 

for  (int  i=0;  i  <  cacheSize-1 ;  ++i)  cout  <<  cache[i]  << 
cout  <<  cache[cacheSize-1 ]  <<  ")\n\n"; 
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cout  <<  "Request\t"; 

for  (int  i=0;  i  <  cacheSize  ++i)  cout  <<  "cache  "  <<  i  <<  "\t" 
cout  <<  "cache  miss"  <<  endl; 

int  cntMisses  =  0 

for(int  i=0;  i<requestLength  ++i) 

{ 

bool  isMiss  =  updateCache(i,  selectedStrategy [strat ])  ; 
if  (isMiss)  ++cntMisses 

cout  <<  "  "  <<  request[i]  <<  "\t"; 

for  (int  1=0;  1  <  cacheSize  ++1)  cout  <<  "  "  <<  cache[l]  <<  "\t"; 

cout  <<  (isMiss  ?  "x"  ;  "")  <<  endl: 

} 

cout<<  "\nTotal  cache  misses:  "  <<  cntMisses  <<  endl; 

} 

for(int  i=0;  i<5;  ++i)  delete  selectedStrategy [ i]  ; 


The  basic  idea  is  simple:  for  every  request  I  have  two  calls  two  my  strategy: 

1 .  apply:  The  strategy  has  to  tell  the  caller  which  page  to  use 

2.  update:  After  the  caller  uses  the  place,  it  tells  the  strategy  whether  it  was  a  miss  or  not.  Then  the  strategy 
may  update  its  internal  data.  The  strategy  LFU  for  example  has  to  update  the  hit  frequency  for  the  cache 
pages,  while  the  LFD  strategy  has  to  recalculate  the  distances  for  the  cache  pages. 

Now  lets  look  of  example  implementations  for  our  five  strategies: 

FIFO 

class  FIFO  public  Strategy  { 

public 

FIFO( )  :  St rategy ( " FIFO" ) 

{ 

for  (int  i=0;  i<cacheSize  ++i)  age[i  =  0; 

} 

int  apply(int  requestlndex)  override 

{ 

int  oldest  =  0; 

for (int  i=0;  i<cacheSize  ++i) 

{ 

if(cache[i  ==  request [ requestlndex  ] ) 

return  i 

else  if(age[i  >  age[oldest]) 
oldest  =  i 

} 

return  oldest; 

} 

void  update(int  cachePos,  int  requestlndex,  bool  cacheMiss)  override 

{ 

//  nothing  changed  we  don't  need  to  update  the  ages 
if ( ! cacheMiss) 

return 
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//  all  old  pages  get  older,  the  new  one  get  0 
for(int  i=0;  i<cacheSize  ++i) 

{ 

if(i  !=  cachePos) 
age [ i ] ++ ; 

else 

age [ i ]  =  0; 

} 

} 

private 

int  age[cacheSize 

}; 

FIFO  just  needs  the  information  how  long  a  page  is  in  the  cache  (and  of  course  only  relative  to  the  other  pages).  So 
the  only  thing  to  do  is  wait  for  a  miss  and  then  make  the  pages,  which  where  not  evicted  older.  For  our  example 
above  the  program  solution  is: 


Strategy  FIFO 
Cache  initial:  (a,b,c) 


Request 

cache  0 

cache  1 

cache  2 

cache  miss 

a 

a 

b 

c 

a 

a 

b 

c 

d 

d 

b 

c 

X 

e 

d 

e 

c 

X 

b 

d 

e 

b 

X 

b 

d 

e 

b 

a 

a 

e 

b 

X 

c 

a 

c 

b 

X 

f 

a 

c 

f 

X 

d 

d 

c 

f 

X 

e 

d 

e 

f 

X 

a 

d 

e 

a 

X 

f 

f 

e 

a 

X 

b 

f 

b 

a 

X 

e 

f 

b 

e 

X 

c 

c 

b 

e 

X 

Total  cache  misses:  13 

Thats  exact  the  solution  from  above. 

LIFO 

class  LIFO  public  Strategy  { 

public 

LIFO( )  :  Strategy( "LIFO" ) 

{ 

for  (int  i=0 ;  i<cacheSize  ++i)  age[i  =  0; 

} 

int  apply(int  requestlndex)  override 

{ 

int  newest  =  0; 

for (int  i=0;  i<cacheSize  ++i) 

{ 

if(cache[ii  ==  request [ requestlndex ] ) 
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return  i 


else  if(age[i  <  age[newest]) 
newest  =  i 

} 

return  newest; 

} 

void  update(int  cachePos,  int  requestlndex,  bool  cacheMiss)  override 

{ 

//  nothing  changed  we  don't  need  to  update  the  ages 
if ( ! cacheMiss) 

return 

//  all  old  pages  get  older,  the  new  one  get  0 
for (int  i=0;  i<cacheSize  ++i) 

{ 

if(i  !=  cachePos) 
age[i]++; 


} 


else 

age [ i ]  =  0; 


private 

int  age[cacheSize 

}; 


The  implementation  of  LIFO  is  more  or  less  the  same  as  by  FIFO  but  we  evict  the  youngest  not  the  oldest  page.  The 
program  results  are: 


Strategy  LIFO 
Cache  initial:  (a,b,c) 


Request 

cache  0 

cache  1 

cache  2 

cache  miss 

a 

a 

b 

c 

a 

a 

b 

c 

d 

d 

b 

c 

X 

e 

e 

b 

c 

X 

b 

e 

b 

c 

b 

e 

b 

c 

a 

a 

b 

c 

X 

c 

a 

b 

c 

f 

f 

b 

c 

X 

d 

d 

b 

c 

X 

e 

e 

b 

c 

X 

a 

a 

b 

c 

X 

f 

f 

b 

c 

X 

b 

f 

b 

c 

e 

e 

b 

c 

X 

c 

e 

b 

c 

Total  cache  misses:  9 

LRU 

class  LRU  public  Strategy  { 

public 

LRU()  :  Strategy( "LRU" ) 
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{ 

for  (int  i=0;  i<cacheSize  ++i)  age[i]  =  0; 

} 

//  here  oldest  mean  not  used  the  longest 

int  apply(int  requestlndex)  override 

{ 

int  oldest  =  0; 

for (int  1=0 ;  i<cacheSize  ++i) 

{ 

if(cache[ij  ==  request [ requestlndex ] ) 

return  i 

else  if(age[i  >  age[oldest]) 
oldest  =  i 

} 

return  oldest; 

} 

void  update(int  cachePos,  int  requestlndex,  bool  cacheMiss)  override 

{ 

//  all  old  pages  get  older,  the  used  one  get  0 
for (int  i=0;  i<cacheSize  ++i) 

{ 

if(i  !=  cachePos) 
age [ i ] ++ ; 


} 


else 

age [ i ]  =  0; 


private 

int  age[cacheSize 

}; 


In  case  of  LRU  the  strategy  is  independent  from  what  is  at  the  cache  page,  its  only  interest  is  the  last  usage.  The 
programm  results  are: 

Strategy  LRU 


Cache  initial:  (a,b,c) 


Request 

cache  0 

cache  1 

cache  2 

cache  miss 

a 

a 

b 

c 

a 

a 

b 

c 

d 

a 

d 

c 

X 

e 

a 

d 

e 

X 

b 

b 

d 

e 

X 

b 

b 

d 

e 

a 

b 

a 

e 

X 

c 

b 

a 

c 

X 

f 

f 

a 

c 

X 

d 

f 

d 

c 

X 

e 

f 

d 

e 

X 

a 

a 

d 

e 

X 

f 

a 

f 

e 

X 

b 

a 

f 

b 

X 

e 

e 

f 

b 

X 
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c  e  c  b  x 

Total  cache  misses:  13 

LFU 

class  LFU  public  Strategy  { 

public 

LFU()  :  Strategy( "LFU" ) 

{ 

for  (int  i=0;  i<cacheSize  ++i)  requestFrequency [ i  =  0; 

} 

int  apply(int  requestlndex)  override 

{ 

int  least  =  0; 

for (int  i=0;  i<cacheSize  ++i) 

{ 

if(cache[i  ==  request! requestlndex] ) 

return  i 

else  if ( requestFrequency [ i  <  requestFrequency! least] ) 
least  =  i 

} 

return  least; 

} 

void  update(int  cachePos,  int  requestlndex,  bool  cacheMiss)  override 

{ 

if (cacheMiss) 

requestFrequency [ cachePos ]  =  1; 

else 

++requestFrequency[ cachePos ] ; 

} 

private 

//  how  frequently  was  the  page  used 

int  requestFrequency [ cacheSize 

}; 

LFU  evicts  the  page  uses  least  often.  So  the  update  strategy  is  just  to  count  every  access.  Of  course  after  a  miss  the 
count  resets.  The  program  results  are: 

Strategy  LFU 


Cache  initial  (a,b,c) 


Request 

cache  0 

cache  1 

cache  2 

cache  miss 

a 

a 

b 

c 

a 

a 

b 

c 

d 

a 

d 

c 

X 

e 

a 

d 

e 

X 

b 

a 

b 

e 

X 

b 

a 

b 

e 

a 

a 

b 

e 

c 

a 

b 

c 

X 

f 

a 

b 

f 

X 

d 

a 

b 

d 

X 
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e  a  b 

a  a  b 

fab 
b  a  b 

e  a  b 

cab 

Total  cache  misses:  10 

LFD 

class  LFD  public  Strategy 
public 

LFD()  :  Strategy( "LFD" ) 

{ 

//  precalc  next  usage  before  starting  to  fullfill  requests 

for  (int  i=0;  i<cacheSize  ++i)  nextUse[i]  =  calcNextUse( -1 ,  cache[i]); 

} 

int  apply(int  requestlndex)  override 

{ 

int  latest  =  0; 

for (int  i=0;  i<cacheSize ;  ++i) 

{ 

if(cache[i]  ==  request! requestlndex] ) 

return  i 

else  if (nextUse [ i ]  >  nextUse[ latest ] ) 
latest  =  i 

} 

return  latest; 

} 

void  update(int  cachePos,  int  requestlndex,  bool  cacheMiss)  override 

{ 

nextUse[cachePos]  =  calcNextUse( requestlndex,  cache[cachePos] ) ; 

} 

private 

int  calcNextUse(int  requestPosition ,  char  pageltem) 

{ 

for(int  i  =  requestPosition+1  ;  i  <  requestLength ;  ++i) 

{ 

if  (request[i  ==  pageltem) 

return  i 

} 

return  requestLength  +  1 ; 

} 

//  next  usage  of  page 

int  nextUse[cacheSize 


x 

x 

x 
x 

{ 


e 

e 

f 

f 

e 

c 


The  LFD  strategy  is  different  from  everyone  before.  Its  the  only  strategy  that  uses  the  future  requests  for  its 
decission  who  to  evict.  The  implementation  uses  the  function  calcNextUse  to  get  the  page  which  next  use  is 
farthest  away  in  the  future.  The  program  solution  is  equal  to  the  solution  by  hand  from  above: 

Strategy  LFD 
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Cache  initial:  (a,b,c) 


Request 

cache  0 

cache  1 

cache  2 

cache  miss 

a 

a 

b 

c 

a 

a 

b 

c 

d 

a 

b 

d 

X 

e 

a 

b 

e 

X 

b 

a 

b 

e 

b 

a 

b 

e 

a 

a 

b 

e 

c 

a 

c 

e 

X 

f 

a 

f 

e 

X 

d 

a 

d 

e 

X 

e 

a 

d 

e 

a 

a 

d 

e 

f 

f 

d 

e 

X 

b 

b 

d 

e 

X 

e 

b 

d 

e 

c 

c 

d 

e 

X 

Total  cache  misses:  8 


The  greedy  strategy  LFD  is  indeed  the  only  optimal  strategy  of  the  five  presented.  The  proof  is  rather  long  and  can 
be  found  here  or  in  the  book  byjon  Kleinberg  and  Eva  Tardos  (see  sources  in  remarks  down  below). 

Algorithm  vs  Reality 

The  LFD  strategy  is  optimal,  but  there  is  a  big  problem.  Its  an  optimal  offline  solution.  In  praxis  caching  is  usually 
an  online  problem,  that  means  the  strategy  is  useless  because  we  cannot  now  the  next  time  we  need  a  particular 
item.  The  other  four  strategies  are  also  online  strategies.  For  online  problems  we  need  a  general  different 
approach. 

Section  11.2:  Ticket  automat 

First  simple  Example: 

You  have  a  ticket  automat  which  gives  exchange  in  coins  with  values  1 , 2,  5, 1 0  and  20.  The  dispension  of  the 
exchange  can  be  seen  as  a  series  of  coin  drops  until  the  right  value  is  dispensed.  We  say  a  dispension  is  optimal 
when  its  coin  count  is  minimal  for  its  value. 


Let  M  in  [  1 , 50]  be  the  price  for  the  ticket  T  and  P  in  [1 ,  50]  the  money  somebody  paid  for  T,  with  P  >=  M.  Let  D=P  M. 
We  define  the  benefit  of  a  step  as  the  difference  between  D  and  D-c  with  c  the  coin  the  automat  dispense  in  this 
step. 

The  Greedy  Technique  for  the  exchange  is  the  following  pseudo  algorithmic  approach: 


Step  1:  while  D 
Step  2:  while  D 
Step  3:  while  D 
Step  4:  while  D 
Step  5:  while  D 


>  20  dispense  a  20  coin  and  set  D  = 

>  10  dispense  a  10  coin  and  set  D  = 

>  5  dispense  a  5  coin  and  set  D  =  D 

>  2  dispense  a  2  coin  and  set  D  =  D 

>  1  dispense  a  1  coin  and  set  D  =  D 


D  -  20 
D  -  10 

-  5 

-  2 
-  1 


Afterwards  the  sum  of  all  coins  clearly  equals  D.  Its  a  greedy  algorithm  because  after  each  step  and  after  each 
repitition  of  a  step  the  benefit  is  maximized.  We  cannot  dispense  another  coin  with  a  higher  benefit. 


Now  the  ticket  automat  as  program  (in  C++): 
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#include  <iostream> 

#include  <vector> 

#include  <string> 

#include  <algorithm> 

using  namespace  std ; 

//  read  some  coin  values,  sort  them  descending, 

//  purge  copies  and  guaratee  the  1  coin  is  in  it 
std : : vector<unsigned  int>  readInCoinValues( ) ; 

int  main() 

{ 

std : : vector<unsigned  int>  coinValues; 
int  ticketPrice 
int  paidMoney 

//  generate  coin  values 

coinValues  =  readInCoinValues( ) ; 

cout  <<  "ticket  price: 
cin  >>  ticketPrice 

cout  <<  "money  paid: 
cin  >>  paidMoney 

if(paidMoney  <=  ticketPrice) 

{ 

cout  <<  "No  exchange  money"  <<  endl; 
return  1 ; 

} 

int  diffValue  =  paidMoney  -  ticketPrice 

//  Here  starts  greedy 

//  we  save  how  many  coins  we  have  to  give  out 
std  :  :  vector<unsigned  int>  coinCount; 

for(auto  coinValue  =  coinValues . begin( ) ; 

coinValue  !=  coinValues . end( ) ;  ++coinValue) 

{ 

int  countCoins  =  0; 

while  (diffValue  >=  *coinValue) 

{ 

diffValue  -=  *coinValue 
countCoins++; 

} 

coinCount . push_ back (count Co ins) ; 

} 

//  print  out  result 

cout  <<  "the  difference  "  <<  paidMoney  -  ticketPrice 
<<  "  is  paid  with:  "  <<  endl; 

for(unsigned  int  i=0,  i  <  coinValues. size() ;  ++i) 

{ 

if (coinCount[ i]  >  0) 

cout  <<  coinCount[i]  <<  "  coins  with  value  " 
<<  coinValues[ i]  <<  endl; 


//  Array  of  coin  values  ascending 
//  M  in  example 
//  P  in  example 
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} 


return  0; 

} 

std  :  : vector<unsigned  int>  readInCoinValues( ) 

{ 

//  coin  values 

std  :  : vector<unsigned  int>  coinValues; 

//  make  sure  1  is  in  vectore 
coinValues . push _back(  1 ) ; 

//  read  in  coin  values  (attention:  error  handling  is  omitted) 
while(true) 

{ 

int  coinValue 

cout  <<  "Coin  value  (<1  to  stop): 
cin  >>  coinValue 

if(coinValue  >  0) 

coinValues . push _back( coin Value ) ; 


else 

break ; 

> 

//  sort  values 

sort (coinValues . beg in( ) ,  coinValues . end( ) ,  std :  :greater<int>( ) ) ; 

//  erase  copies  of  same  value 

auto  last  =  std : : unique(coinValues .  begin( ) ,  coinValues . end( )) ; 
coinValues . erase( last ,  coinValues . end ()) ; 

//  print  array 
cout  <<  "Coin  values: 

for(auto  i  coinValues) 
cout  <<  i  <<  "  " ; 

cout  <<  endl; 

return  coinValues; 

} 

Be  aware  there  is  now  input  checking  to  keep  the  example  simple.  One  example  output: 

Coin  value  (<1  to  stop) :  2 
Coin  value  (<1  to  stop) :  4 
Coin  value  (<1  to  stop) :  7 
Coin  value  (<1  to  stop) :  9 
Coin  value  (<1  to  stop) :  14 
Coin  value  (<1  to  stop)  :  4 
Coin  value  (<1  to  stop) :  0 
Coin  values:  14  9  7  4  2  1 
ticket  price:  34 
money  paid:  67 

the  difference  33  is  paid  with: 

2  coins  with  value  14 
1  coins  with  value  4 
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1  coins  with  value 


As  long  as  1  is  in  the  coin  values  we  now,  that  the  algorithm  will  terminate,  because: 

•  D  strictly  decreases  with  every  step 

•  D  is  never  >0  and  smaller  than  than  the  smallest  coin  1  at  the  same  time 
But  the  algorithm  has  two  pitfalls: 

1 .  Let  C  be  the  biggest  coin  value.  The  runtime  is  only  polynomial  as  long  as  D/C  is  polynomial,  because  the 
representation  of  D  uses  only  log  D  bits  and  the  runtime  is  at  least  linear  in  D/C. 

2.  In  every  step  our  algorithm  chooses  the  local  optimum.  But  this  is  not  sufficient  to  say  that  the  algorithm 
finds  the  global  optimal  solution  (see  more  information  here  or  in  the  Book  of  Korte  and  Vvgen). 

A  simple  counter  example:  the  coins  are  1,3,4  and  D=6.  The  optimal  solution  is  clearly  two  coins  of  value  3  but 
greedy  chooses  4  in  the  first  step  so  it  has  to  choose  1  in  step  two  and  three.  So  it  gives  no  optimal  soution.  A 
possible  optimal  Algorithm  for  this  example  is  based  on  dynamic  programming. 

Section  11.3:  Interval  Scheduling 

We  have  a  set  of  jobs  J={a,b,c,d,e,f,g}.Letj  in  Jbea  job  than  its  start  at  sj  and  ends  at  fj.  Two  jobs  are 
compatible  if  they  don't  overlap.  A  picture  as  example: 


Jobs  e  and  g  are 
incompatible 

b 


e 


d 


a 


c 


g 


The  goal  is  to  find  the  maximum  subset  of  mutually  compatible  jobs.  There  are  several  greedy  approaches  for 
this  problem: 

1 .  Earliest  start  time:  Consider  jobs  in  ascending  order  of  sj 

2.  Earliest  finish  time:  Consider  jobs  in  ascending  order  of  f  j 

3.  Shortest  interval:  Consider  jobs  in  ascending  order  of  f  j  -s  j 

4.  Fewest  conflicts:  For  each  job  j,  count  the  number  of  conflicting  jobs  cj 

The  question  now  is,  which  approach  is  really  successful!.  Early  start  time  definetly  not,  here  is  a  counter  example 


GoalKicker.com  -  Algorithms  Notes  for  Professionals 


72 


Shortest  interval  is  not  optimal  either 


and  fewest  conflicts  may  indeed  sound  optimal,  but  here  is  a  problem  case  for  this  approach: 


Which  leaves  us  with  earliest  finish  time.  The  pseudo  code  is  quiet  simple: 

1 .  Sort  jobs  by  finish  time  so  that  fi  <=f2<= . .  ,<=fn 

2.  Let  A  be  an  empty  set 

3.  for  j=l  to  n  if  j  is  compatible  to  all  jobs  in  A  set  A=A+{  j } 

4.  A  is  a  maximum  subset  of  mutually  compatible  jobs 

Or  as  C++  program: 

#include  <iostream> 

#include  <utility> 

#include  <tuple> 

#include  <vector> 

#include  <algorithm> 

const  int  jobCnt  =  10: 

//  Job  start  times 

const  int  startTimes[]  =  {  2,  3,  1,  4,  3,  2,  6,  7,  8,  9}; 

//  Job  end  times 

const  int  endTimes[]  =  {  4,  4,  3,  5,  5,  5,  8,  9,  9,  10}; 

using  namespace  std ; 

int  main() 

{ 

vector<pair<int , int>>  jobs; 
for(int  i=0;  i< jobCnt;  ++i) 

jobs.push_back(make_pair(startTimes[i] ,  endTimes[i] )) ; 

//  step  1 :  sort 

sort( jobs . begin( ) ,  jobs.end() , [ ] (pair<int, int>  pi,  pair<int , int>  p2) 

{  return  pi. second  <  p2. second;  }); 


//  step  2:  empty  set  A 
vector<int>  A 

//  step  3 : 

for(int  i=0;  i< jobCnt;  ++i) 

{ 
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auto  job  =  jobs [ i ] ; 
bool  isCompatible  =  true; 

for(auto  joblndex  A) 

{ 

//  test  whether  the  actual  job  and  the  job  from  A  are  incompatible 
if ( job . second  >=  jobs[ joblndex] .first  && 
job. first  <=  jobs[ joblndex] .second) 

{ 

isCompatible  =  false; 
break ; 

> 

} 

if ( isCompatible) 

A.push_back(i) ; 

} 

//step  4:  print  A 
cout  <<  "Compatible: 

for(auto  i  A) 

cout  <<  "("  <<  jobs [ i ]. first  <<  <<  jobs [ i ]. second  <<  ")  "; 

cout  <<  endl; 

return  0; 

} 

The  output  for  this  example  is:  Compatible :  (1,3)  (4,5)  (6,8)  (9,10) 

The  implementation  of  the  algorithm  is  clearly  in  0(nA2).  There  is  a  0(n  log  n)  implementation  and  the  interested 
reader  may  continue  reading  below  (Java  Example). 

Now  we  have  a  greedy  algorithm  for  the  interval  scheduling  problem,  but  is  it  optimal? 

Proposition:  The  greedy  algorithm  earliest  finish  time  is  optimal. 

Proof:  (fay  contradiction) 

Assume  greedy  is  not  optimal  and  il ,  i2,  . . . ,  ik  denote  the  set  of  jobs  selected  by  greedy.  Let  jl ,  j2 . jm 

denote  the  set  of  jobs  in  an  optimal  solution  with  il  =  jl ,  i2=j2,  . . . ,  ir=j  r  for  the  largest  possible  value  of  r. 

The  job  i(  r+1 )  exists  and  finishes  before  j  ( r+1 )  (earliest  finish).  But  than  is  jl ,  j 2 ,  . .  . ,  j  r,  i(  r+1 ) ,  j  ( r+2) ,  .  . . ,  jm 
also  a  optimal  solution  and  for  all  k  in  [  1 ,  ( r+1 )  ]  is  jk=ik.  thats  a  contradiction  to  the  maximality  of  r.  This 
concludes  the  proof. 

This  second  example  demonstrates  that  there  are  usually  many  possible  greedy  strategies  but  only  some  or  even 
none  might  find  the  optimal  solution  in  every  instance. 

Below  is  a  Java  program  that  runs  in  0(n  log  n) 

import  java . util . Arrays ; 
import  java . util . Comparator ; 

class  Job 

{ 

int  start,  finish,  profit; 

Job(int  start,  int  finish,  int  profit) 
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{ 

this. start  =  start; 
this. finish  =  finish 
this. profit  =  profit; 

} 

} 


class  JobComparator  implements  Comparator<Job> 

{ 

public  int  compare(Job  a,  Job  b) 

{ 

return  a. finish  <  b. finish  ?  -1  :  a. finish  ==  b. finish  ?  0  :  1; 

} 

} 

public  class  WeightedlntervalScheduling 

{ 

static  public  int  binarySearch( Job  jobs[  ,  int  index) 

{ 

int  lo  =0,  hi  =  index  -  1 ; 

while  (lo  <=  hi) 

{ 

int  mid  =  (lo  +  hi)  /  2; 

if  (jobs[mid] .finish  <=  jobs[index] .start) 

{ 

if  (jobs[mid  +  I  ]. finish  <=  jobs[index] .start) 
lo  =  mid  +  1 ; 

else 

return  mid; 

} 

else 

hi  =  mid  -  1 ; 

} 

return  -1 ; 

} 

static  public  int  schedule(Job  jobs[]) 

{ 

Arrays. sort( jobs,  new  JobComparator( ) ) ; 

int  n  =  jobs. length; 
int  table[]  =  new  int[n]; 
table[0]  =  jobs [0] . prof it ; 

for  (int  i=1 ;  i<n  i++) 

{ 

int  inclProf  =  jobs[i  .profit; 
int  1  =  binarySearch( jobs,  i) ; 
if  (1  ! =  -1) 

inclProf  +=  table[l]; 

table[i]  =  Math .max(inclProf ,  table[i-1 ] ) ; 

} 

return  table[n-1 ] ; 

} 

public  static  void  main(String[ ]  args) 

{ 
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Job  jobs[]  =  {new  Job(1,  2,  50),  new  Job(3,  5,  20), 

new  Job(6,  19,  100),  new  Job(2,  100,  200)}; 

System. out . println( "Optimal  profit  is  "  +  schedule) jobs) ) ; 

} 

} 

And  the  expected  output  is: 

Optimal  profit  is  250 

Section  11.4:  Minimizing  Lateness 

There  are  numerous  problems  minimizing  lateness,  here  we  have  a  single  resource  which  can  only  process  one  job 
at  a  time.  Job  j  requires  tj  units  of  processing  time  and  is  due  at  time  dj.  if  j  starts  at  time  sj  it  will  finish  at  time 
f  j=sj+t  j.  We  define  lateness  L=max{0,  f  j-dh}  for  all  j.  The  goal  is  to  minimize  the  maximum  lateness  L. 

1  234  5  6 

tj  321  4  3  2 
dj  68991011 

Job  3  2  2  55  5  444  4  1  1  1  6  6 

Time  1  23456789101112131415 
Lj  -8-5-4  1  7  4 

The  solution  L=7  is  obviously  not  optimal.  Lets  look  at  some  greedy  strategies: 

1 .  Shortest  processing  time  first:  schedule  jobs  in  ascending  order  og  processing  time  j ' 

2.  Earliest  deadline  first:  Schedule  jobs  in  ascending  order  of  deadline  d  j 

3.  Smallest  slack:  schedule  jobs  in  ascending  order  of  slack  d  j  -t  j 

Its  easy  to  see  that  shortest  processing  time  first  is  not  optimal  a  good  counter  example  is 

1  2 

tj  1  5 
dj  105 

the  smallest  stack  solution  has  simillar  problems 

1  2 

tj  1  5 
dj  3  5 

the  last  strategy  looks  valid  so  we  start  with  some  pseudo  code: 

1 .  Sort  n  jobs  by  due  time  so  that  dl  <=d2<= .  . .  <=dn 

2.  Sett=0 

3.  for  j  =  1  to  n 

°  Assign  job  j  to  interval  [  t ,  t+t  j  ] 
o  set  sj=t  and  f  j=t+t  j 
°  set  t=t+t j 

4.  return  intervals  [  si ,  fl  ],[  s2 ,  f 2 ],...,[  sn ,  fn  ] 

And  as  implementation  in  C++: 

#include  <iostream> 
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#include  <utility> 

#include  <tuple> 

#include  <vector> 

#include  <algorithm> 

const  int  jobCnt  =  10; 

//  Job  start  times 

const  int  processTimes[ ]  =  {  2,  3,  1,  4,  3,  2,  3,  5,  2,  1 } ; 

//  Job  end  times 

const  int  dueTimes[]  =  {  4,  7,  9,  13,  8,  17,  9,  11,  22,  25}; 

using  namespace  std ; 

int  main() 

{ 

vector<pair<int , int>>  jobs; 
for(int  i=0;  i<jobCnt;  ++i) 

jobs.push_back(make_pair(processTimes[i] ,  dueTimes[i] )) ; 

//  step  1  :  sort 

sort( jobs.begin() ,  jobs.end() , [ ] (pair<int, int>  pi,  pair<int , int>  p2) 

{  return  pi .second  <  p2. second;  }); 


//  step  2 :  set  t=0 
int  t  =  0 ; 


//  step  3 : 

vector<pair<int , int>>  joblntervals; 

for(int  i=0,  i< jobCnt;  ++i) 

{ 

joblntervals . push_back(make_pair(t ,  t+jobs [ i ] . first) ) ; 
t  +=  jobs [ i ] . first ; 

} 

//step  4:  print  intervals 
cout  <<  "Intervals : \n"  <<  endl 


int  lateness  =  0 


for(int  i=0;  i<jobCnt;  ++i) 

{ 

auto  pair  =  jobIntervals[ i] ; 

lateness  =  max(lateness,  pair. second- jobs [ i ] .second); 


} 


cout  << 
<< 


"("  <<  pair. first  <<  ","  <<  pair. second  <<  ")  " 
"Lateness:  "  <<  pair . second- jobs[ i] . second  <<  std::endl; 


cout  <<  "\nmaximal  lateness  is  "  <<  lateness  <<  endl; 


return  0; 

} 


And  the  output  for  this  program  is: 

Intervals : 
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(0,2)  Lateness:-2 
(2,5)  Lateness:-2 

(5.8)  Lateness:  0 

(8.9)  Lateness:  0 
(9,12)  Lateness:  3 
(12,17)  Lateness:  6 
(17,21)  Lateness:  8 
(21 ,23)  Lateness :  6 

(23.25)  Lateness:  3 

(25.26)  Lateness:  1 

maximal  lateness  is  8 

The  runtime  of  the  algorithm  is  obviously  0(n  log  n)  because  sorting  is  the  dominating  operation  of  this  algorithm. 
Now  we  need  to  show  that  it  is  optimal.  Clearly  an  optimal  schedule  has  no  idle  time,  the  earliest  deadline  first 
schedule  has  also  no  idle  time. 

Lets  assume  the  jobs  are  numbered  so  that  dl  <=d2<= . .  .<=dn.  We  say  a  inversion  of  a  schedule  is  a  pair  of  jobs  i 
and  j  so  that  i<  j  but  j  is  scheduled  before  i.  Due  to  its  definition  the  earliest  deadline  first  schedule  has  no 
inversions.  Of  course  if  a  schedule  has  an  inversion  it  has  one  with  a  pair  of  inverted  jobs  scheduled  consecutively. 

Proposition:  Swapping  two  adjacent,  inverted  jobs  reduces  the  number  of  inversions  by  one  and  does  not 
increase  the  maximal  lateness. 

Proof:  Let  L  be  the  lateness  before  the  swap  and  M  the  lateness  afterwards.  Because  exchanging  two  adjacent  jobs 
does  not  move  the  other  jobs  from  their  position  it  is  Lk=Mk  for  all  k  !  =  i,  j. 

Clearly  it  is  Mi<=Li  since  job  i  got  scheduled  earlier,  if  job  j  is  late,  so  follows  from  the  definition: 

Mj  =  fi-dj  (definition) 

<=  fi-di  (since  i  and  j  are  exchanged) 

<=  Li 

That  means  the  lateness  after  swap  is  less  or  equal  than  before.  This  concludes  the  proof. 

Proposition:  The  earliest  deadline  first  schedule  S  is  optimal. 

Proof:  (by  contradiction) 

Lets  assume  S*  is  optimal  schedule  with  the  fewest  possible  number  of  inversions,  we  can  assume  that  S*  has  no 
idle  time.  If  S*  has  no  inversions,  then  S=S*  and  we  are  done.  If  S*  has  an  inversion,  than  it  has  an  adjacent 
inversion.  The  last  Proposition  states  that  we  can  swap  the  adjacent  inversion  without  increasing  lateness  but  with 
decreasing  the  number  of  inversions.  This  contradicts  the  definition  of  S*. 

The  minimizing  lateness  problem  and  its  near  related  minimum  makespan  problem,  where  the  question  for  a 
minimal  schedule  is  asked  have  lots  of  applications  in  the  real  world.  But  usually  you  don't  have  only  one  machine 
but  many  and  they  handle  the  same  task  at  different  rates.  These  problems  get  NP-complete  really  fast. 

Another  interesting  question  arises  if  we  don't  look  at  the  offline  problem,  where  we  have  all  tasks  and  data  at 
hand  but  at  the  online  variant,  where  tasks  appear  during  execution. 
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Chapter  12:  Prim's  Algorithm 

Section  12.1:  Introduction  To  Prim's  Algorithm 

Let's  say  we  have  8  houses.  We  want  to  setup  telephone  lines  between  these  houses.  The  edge  between  the  houses 
represent  the  cost  of  setting  line  between  two  houses. 


Our  task  is  to  set  up  lines  in  such  a  way  that  all  the  houses  are  connected  and  the  cost  of  setting  up  the  whole 
connection  is  minimum.  Now  how  do  we  find  that  out?  We  can  use  Prim's  Algorithm. 

Prim's  Algorithm  is  a  greedy  algorithm  that  finds  a  minimum  spanning  tree  for  a  weighted  undirected  graph.  This 
means  it  finds  a  subset  of  the  edges  that  forms  a  tree  that  includes  every  node,  where  the  total  weight  of  all  the 
edges  in  the  tree  are  minimized.  The  algorithm  was  developed  in  1 930  by  Czech  mathematician  Vojtech  larnik  and 
later  rediscovered  and  republished  by  computer  scientist  Robert  Clay  Prim  in  1 957  and  EdsgerWvbe  Dijkstra  in 
1 959.  It  is  also  known  as  DJP  algorithm,  Jarnik's  algorithm,  Prim-Jarnik  algorithm  or  Prim-Dijsktra  algorithm. 

Now  let's  look  at  the  technical  terms  first.  If  we  create  a  graph,  S  using  some  nodes  and  edges  of  an  undirected 
graph  G,  then  S  is  called  a  subgraph  of  the  graph  G.  Now  S  will  be  called  a  Spanning  Tree  if  and  only  if: 

•  It  contains  all  the  nodes  of  G. 

•  It  is  a  tree,  that  means  there  is  no  cycle  and  all  the  nodes  are  connected. 

•  There  are  (n-1)  edges  in  the  tree,  where  n  is  the  number  of  nodes  in  G. 

There  can  be  many  Spanning  Tree's  of  a  graph.  The  Minimum  Spanning  Tree  of  a  weighted  undirected  graph  is  a 
tree,  such  that  sum  of  the  weight  of  the  edges  is  minimum.  Now  we'll  use  Prim's  algorithm  to  find  out  the 
minimum  spanning  tree,  that  is  how  to  set  up  the  telephone  lines  in  our  example  graph  in  such  way  that  the  cost  of 
set  up  is  minimum. 

At  first  we'll  select  a  source  node.  Let's  say,  node-1  is  our  source.  Now  we'll  add  the  edge  from  node-1  that  has  the 
minimum  cost  to  our  subgraph.  Here  we  mark  the  edges  that  are  in  the  subgraph  using  the  color  blue.  Here  1-5  is 
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1 


2 


1 


our  desired  edge. 

Now  we  consider  all  the  edges  from  node-1  and  node-5  and  take  the  minimum.  Since  1-5  is  already  marked,  we 


take  1-2. 

This  time,  we  consider  node-1,  node-2  and  node-5  and  take  the  minimum  edge  which  is  5-4. 
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The  next  step  is  important.  From  node-1,  node-2,  node-5  and  node-4,  the  minimum  edge  is  2-4.  But  if  we  select 
that  one,  it'll  create  a  cycle  in  our  subgraph.  This  is  because  node-2  and  node-4  are  already  in  our  subgraph.  So 
taking  edge  2-4  doesn't  benefit  us.  We'll  select  the  edges  in  such  way  that  it  adds  a  new  node  in  our  subgraph.  So  we 


select  edge  4-8. 

If  we  continue  this  way,  we'll  select  edge  8-6,  6-7  and  4-3.  Our  subgraph  will  look  like: 
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This  is  our  desired  subgraph,  that'll  give  us  the  minimum  spanning  tree.  If  we  remove  the  edges  that  we  didn't 


select,  we'll  get: 

This  is  our  minimum  spanning  tree  (MST).  So  the  cost  of  setting  up  the  telephone  connections  is:  4  +  2  +  5  +  11+9 
+  2  +  1  =  34.  And  the  set  of  houses  and  their  connections  are  shown  in  the  graph.  There  can  be  multiple  MST  of  a 
graph.  It  depends  on  the  source  node  we  choose. 

The  pseudo-code  of  the  algorithm  is  given  below: 

Procedure  PrimsMST(Graph)  :  //  here  Graph  is  a  non-empty  connected  weighted  graph 

Vnew[ ]  =  {x}  //  New  subgraph  Vnew  with  source  node  x 
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Enew[ ]  =  {} 

while  Vnew  is  not  equal  to  V 
u  ->  a  node  from  Vnew 

v  ->  a  node  that  is  not  in  Vnew  such  that  edge  u  v  has  the  minimum  cost 

//  if  two  nodes  have  same  weight,  pick  any  of  them 

add  v  to  Vnew 

add  edge  (u,  v)  to  Enew 

end  while 

Return  Vnew  and  Enew 

Complexity: 

Time  complexity  of  the  above  naive  approach  is  0(V2).  It  uses  adjacency  matrix.  We  can  reduce  the  complexity 
using  priority  queue.  When  we  add  a  new  node  to  Vnew,  we  can  add  its  adjacent  edges  in  the  priority  queue.  Then 
pop  the  minimum  weighted  edge  from  it.  Then  the  complexity  will  be:  O(ElogE),  where  E  is  the  number  of  edges. 
Again  a  Binary  Heap  can  be  constructed  to  reduce  the  complexity  to  O(ElogV). 

The  pseudo-code  using  Priority  Queue  is  given  below: 

Procedure  MSTPrim(Graph ,  source) : 
for  each  u  in  V 
key[u]  :=  inf 
parent[u]  :=  NULL 

end  for 

key[source  :=  0 
Q  =  Priority_Queue( ) 

Q  =  V 

while  Q  is  not  empty 
u  ->  Q.pop 

for  each  v  adjacent  to  i 

if  v  belongs  to  Q  and  Edge(u,v)  <  key[v|  //  here  Edge(u,  v)  represents 

//  cost  of  edge(u,  v) 

parent[v]  :=  u 
key[v]  :=  Edge(u,  v) 
end  if 

end  for 
end  while 

Here  key[]  stores  the  minimum  cost  of  traversing  node-v.  parent[]  is  used  to  store  the  parent  node.  It  is  useful  for 
traversing  and  printing  the  tree. 

Below  is  a  simple  program  in  Java: 

import  java. util.*; 

public  class  Graph 

{ 

private  static  int  infinite  =  9999999 ; 
int[ ] [ ]  LinkCost; 
int  NNodes; 

Graph(int[  ]  [  ]  mat) 

{ 

int  i,  j  ; 

NNodes  =  mat. length; 

LinkCost  =  new  int[NNodes] [NNodes]  ; 
for  (  i=0;  i  <  NNodes;  i++) 

{ 

for  (  j =0 ;  j  <  NNodes;  j++) 

{ 
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LinkCost[i] [ j  ]  =  mat[i][j]; 
if  (  LinkCost [ i ] [ j ]  ==  0  ) 

LinkCost [ i  |  [ j ]  =  infinite 

} 

} 

for  (  i=0;  i  <  NNodes  i++) 

{ 

for  (  j=0;  j  <  NNodes;  j++) 

if  (  LinkCost [ i ][ j ]  <  infinite  ) 

System. out .print(  "  "  +  LinkCost[i  [j]  +  "  "  ); 

else 

System .out . print ( "  *  "  ); 

System .out . print In ( ) ; 

} 

} 

public  int  unReached (boolean [ ]  r) 

{ 

boolean  done  =  true 
for  (  int  i  =  0;  i  <  r. length;  i++  ) 
if  (  r[i]  ==  false  ) 
return  i 
return  -1 ; 

} 

public  void  Prim(  ) 

{ 

int  i,  j,  k,  x,  y 

boolean[]  Reached  =  new  boolean[NNodes]  ; 
int [  ]  predNode  =  new  int[NNodes]  ; 

Reached[0]  =  true 

for  (  k  =  1 ;  k  <  NNodes ;  k++  ) 

{ 

Reached[k]  =  false 

} 

predNode[0]  =  0; 
printReachSet(  Reached  ); 
for  (k  =  1 ,  k  <  NNodes;  k++) 

{ 

x  =  y  =  0; 

for  (  i  =  0;  i  <  NNodes;  i++  ) 
for  (  j  =  0;  j  <  NNodes,  j++  ) 

{ 

if  (  Reached[i]  &&  [Reached] j]  && 

LinkCost [ i ][ j  ]  <  LinkCost[x] [y]  ) 

{ 

x  =  i 


y  =  j; 

} 

} 

System .out .println( "Min  cost  edge: 

+  x  +  " , "  + 

+  y  +  ")"  + 

"cost  =  "  + 

predNode[y]  =  x; 

Reached[y]  =  true 
printReachSet]  Reached  ) ; 

System .out . print In ( ) ; 

} 

int [ ]  a=  predNode 
for  (  i  =  0;  i  <  NNodes;  i++  ) 

System. out . println(  a [ i  +  "  --> 

} 

void  printReachSet (boolean [ ]  Reached  ) 


("  + 


LinkCost[x] [y  ) ; 


+  i  ); 
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System. out . print ( "ReachSet 
for  (int  i  =  0;  i  <  Reached 
if  (  Reached [i]  ) 

System. out .print(  i  + 
//System. out ,println() ; 


=  "); 

.length;  i++  ) 
"  "); 


} 


public  static  void  main(String [ ]  args) 

{ 

int[][]  conn  =  {10,3,0,2,0, 

{3, 0,  0,  0,  0, 


{0,  0, 0, 6, 0, 
{2, 0,6, 0,1, 
{0, 0,0,1, 0, 
{0, 0,1,0, 0, 
{0,  0,  0,  0,  0, 
{0,4, 2, 0,0, 
{4,  0,  0,  0,  8, 
}; 

Graph  G  =  new  Graph(conn)  ; 

G . Prim( )  ; 


0, 0, 0,4}, 
0,0, 4,0}, 
1,0, 2,0}, 
0,  0, 0, 0} , 
0,  0, 0, 8}, 
0,  8, 0, 0} , 
8, 0,0,0}, 
0,  0, 0, 0} , 
0,  0, 0, 0} 


} 


} 


//  0 
//  7 
// 

// 

// 

// 


//  6 
//  7 
//  8 


Compile  the  above  code  using  javac  Graph,  java 
Output: 


$  java  Graph 

*  3  *  2  * 

*  k  k 

3  *  * 

*  * 

*  *  4 

■k  k  k 

6  * 

1  *  2 

2*6 

k  1 

it  it  it 

k  k  k 

1  * 

it  it  it 

*  *  i 

*  * 

*  8  * 

k  k  k 

*  * 

8  *  * 

*  4  2 

*  * 

it  it  it 

4  *  * 

k  8 

it  it  it 

ReachSet 

=  0  Min  cost  edc 

ReachSet 

Min  cost 

=  03 

edge 

(3 , 4)cost 

ReachSet 

=  03 

4 

Min  cost 

edge 

(0,1 )cost 

ReachSet 

=  0  1 

3  4 

Min  cost 

edge 

(0, 8)cost 

ReachSet 

=  0  1 

3  4  8 

Min  cost 

edge 

(1 , 7)cost 

ReachSet 

=  0  1 

3  4  7  8 

Min  cost 

edge 

(7, 2)cost 

ReachSet 

=  0  1 

2  3  4  7  8 

Min  cost 

edge 

(2 , 5)cost 

ReachSet 

=  0  1 

2  3  4  5  7 

Min  cost 

edge 

(5, 6)cost 

ReachSet 

=  0  1 

2  3  4  5  6 

-  -> 
-  -  > 
-  -  > 
-  -> 
-  -> 
-  -> 
-  -  > 
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1  -->  7 
0  -->  8 
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Chapter  13:  Bellman-Ford  Algorithm 

Section  13.1:  Single  Source  Shortest  Path  Algorithm  (Given 
there  is  a  negative  cycle  in  a  graph) 

Before  reading  this  example,  it  is  required  to  have  a  brief  idea  on  edge-relaxation.  You  can  learn  it  from  here 

Bellman-Ford  Algorithm  is  computes  the  shortest  paths  from  a  single  source  vertex  to  all  of  the  other  vertices  in  a 
weighted  digraph.  Even  though  it  is  slower  than  Dijkstra's  Algorithm,  it  works  in  the  cases  when  the  weight  of  the 
edge  is  negative  and  it  also  finds  negative  weight  cycle  in  the  graph.  The  problem  with  Dijkstra's  Algorithm  is,  if 
there's  a  negative  cycle,  you  keep  going  through  the  cycle  again  and  again  and  keep  reducing  the  distance  between 
two  vertices. 

The  idea  of  this  algorithm  is  to  go  through  all  the  edges  of  this  graph  one-by-one  in  some  random  order.  It  can  be 
any  random  order.  But  you  must  ensure,  if  u-v  (where  u  and  v  are  two  vertices  in  a  graph)  is  one  of  your  orders, 
then  there  must  be  an  edge  from  u  to  v.  Usually  it  is  taken  directly  from  the  order  of  the  input  given.  Again,  any 
random  order  will  work. 

After  selecting  the  order,  we  will  relax  the  edges  according  to  the  relaxation  formula.  For  a  given  edge  u-v  going 
from  u  to  v  the  relaxation  formula  is: 

if  distanceju'  +  cost[u][v]  <  d[v] 
d [ v ]  =  d [ u ]  +  cost [ u ] [ v ] 

That  is,  if  the  distance  from  source  to  any  vertex  u  +  the  weight  of  the  edge  u-v  is  less  than  the  distance  from 
source  to  another  vertex  v,  we  update  the  distance  from  source  to  v.  We  need  to  relax  the  edges  at  most  (V-1) 
times  where  V  is  the  number  of  edges  in  the  graph.  Why  (V-1)  you  ask?  We'll  explain  it  in  another  example.  Also  we 
are  going  to  keep  track  of  the  parent  vertex  of  any  vertex,  that  is  when  we  relax  an  edge,  we  will  set: 

parentjv]  =  u 

It  means  we've  found  another  shorter  path  to  reach  v  via  u.  We  will  need  this  later  to  print  the  shortest  path  from 
source  to  the  destined  vertex. 

Let's  look  at  an  example.  We  have  a  graph: 


d[l]=0 


d[2]— inf  d[6]—  inf 
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We  have  selected  1  as  the  source  vertex.  We  want  to  find  out  the  shortest  path  from  the  source  to  all  other 
vertices. 

At  first,  d[1]  =  0  because  it  is  the  source.  And  rest  are  infinity,  because  we  don't  know  their  distance  yet. 

We  will  relax  the  edges  in  this  sequence: 

+ - + - + - + - + - + - + - + 

|  Serial  |  1  |  2|  3|  4|  5|  6| 

+ - + - + - + - + - + - + - + 

|  Edge  |  4->5  |  3->4  |  l->3  |  l->4  |  4->6  |  2->3  | 

+ - + - + - + - + - + - + - + 


You  can  take  any  sequence  you  want.  If  we  relax  the  edges  once,  what  do  we  get?  We  get  the  distance  from  source 
to  all  other  vertices  of  the  path  that  uses  at  most  1  edge.  Now  let's  relax  the  edges  and  update  the  values  of  d[].  We 
get: 

1 .  d[4]  +  cost[4][5]  =  infinity  +  7  =  infinity.  We  can't  update  this  one. 

2.  d[2]  +  cost[3][4]  =  infinity.  We  can't  update  this  one. 

3.  d[1]  +  cost[1][3]  =  0  +  2  =  2  <  d[2].  So  d[3]  =  2.  Also  parent[1]  =  1. 

4.  d[1]  +  cost[1][4]  =  4.  So  d[4]  =  4  <  d[4],  parent[4]  =  1. 

5.  d[4]  +  cost[4][6]  =  9.  d[6]  =  9  <  d[6],  parent[6]  =  4. 

6.  d[2]  +  cost[2][3]  =  infinity.  We  can't  update  this  one. 

We  couldn't  update  some  vertices,  because  the  d[u]  +  cost[u][v]  <  d[v]  condition  didn't  match.  As  we  have  said 
before,  we  found  the  paths  from  source  to  other  nodes  using  maximum  1  edge. 


d[2]=mf 


d[6]=9 


Our  second  iteration  will  provide  us  with  the  path  using  2  nodes.  We  get: 

1 .  d[4]  +  cost[4][5]  =  12  <  d[5],  d[5]  =  12.  parent[5]  =  4. 

2.  d[3]  +  cost[3][4]  =  1  <  d[4],  d[4]  =  1.  parent[4]  =  3. 

3.  d[3]  remains  unchanged. 

4.  d[4]  remains  unchanged. 

5.  d[4]  +  cost[4][6]  =  6  <  d[6].  d[6]  =  6.  parent[6]  =  4. 

6.  d[3]  remains  unchanged. 
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Our  graph  will  look  like: 


d[2]=inf 


d[6]=4 


Our  3rd  iteration  will  only  update  vertex  5,  where  d[5]  will  be  8.  Our  graph  will  look  like: 


d[2]=inf 


d[6]=4 


After  this  no  matter  how  many  iterations  we  do,  we'll  have  the  same  distances.  So  we  will  keep  a  flag  that  checks  if 
any  update  takes  place  or  not.  If  it  doesn't,  we'll  simply  break  the  loop.  Our  pseudo-code  will  be: 

Procedure  Bellman  Ford(Graph,  source) : 
n  :=  number  of  vertices  in  Graph 
for  i  f  rom  1  to  n 

d[i  :=  infinity 
parent[i]  :=  NULL 
end  for 

d[source]  :=  0 
for  i  f  rom  1  to  n-1 
flag  :=  false 

for  all  edges  from  (u,v)  in  Graph 
if  d[u]  +  cost[u][v]  <  d[v] 
d[v]  :=  d[u]  +  cost[u][v] 
parent[v]  :=  u 
flag  :=  true 


GoalKicker.com  -  Algorithms  Notes  for  Professionals 


89 


end  if 
end  for 

if  flag  ==  false 
break 

end  for 
Return  d 

To  keep  track  of  negative  cycle,  we  can  modify  our  code  using  the  procedure  described  here.  Our  completed 
pseudo-code  will  be: 

Procedure  Bellman  Ford  With  Negative-Cycle  Detection(Graph,  source): 
n  :=  number  of  vertices  in  Graph 
for  i  f  rom  1  to  n 

d[i  :=  infinity 
parent[i]  :=  NULL 
end  for 

d[source]  :=  0 
for  i  f  rom  1  to  n-1 
flag  :=  false 

for  all  edges  from  (u,v)  in  Graph 
if  d[u]  +  cost[u][v]  <  d[v] 
d[v]  :=  d[u]  +  cost[u][v] 
parent[v]  :=  u 
flag  :=  true 
end  if 
end  for 

if  flag  ==  false 
break 

end  for 

for  all  edges  from  (u,v)  in  Graph 
if  d[u]  +  cost[u][v]  <  d[v] 

Return  "Negative  Cycle  Detected" 
end  if 

end  for 
Return  d 

Printing  Path: 

To  print  the  shortest  path  to  a  vertex,  we'll  iterate  back  to  its  parent  until  we  find  NULL  and  then  print  the  vertices. 
The  pseudo-code  will  be: 

Procedure  PathPrinting(u) 
v  :=  parent[u 

if  v  ==  NULL 
return 

PathPrinting(v) 
print  ->  u 

Complexity: 

Since  we  need  to  relax  the  edges  maximum  (V-1)  times,  the  time  complexity  of  this  algorithm  will  be  equal  to  0(V  * 
E)  where  E  denotes  the  number  of  edges,  if  we  use  ad  j  acency  list  to  represent  the  graph.  However,  if  adjacency 
matrix  is  used  to  represent  the  graph,  time  complexity  will  be  0(VA3).  Reason  is  we  can  iterate  through  all  edges  in 
0(E)  time  when  adjacency  list  is  used,  but  it  takes  0(VA2)  time  when  adjacency  matrix  is  used. 

Section  13.2:  Detecting  Negative  Cycle  in  a  Graph 

To  understand  this  example,  it  is  recommended  to  have  a  brief  idea  about  Bellman-Ford  algorithm  which  can  be  found 


GoalKicker.com  -  Algorithms  Notes  for  Professionals 


90 


here 


Using  Bellman-Ford  algorithm,  we  can  detect  if  there  is  a  negative  cycle  in  our  graph.  We  know  that,  to  find  out  the 
shortest  path,  we  need  to  relax  all  the  edges  of  the  graph  (V-1)  times,  where  V  is  the  number  of  vertices  in  a  graph. 
We  have  already  seen  that  in  this  example,  after  (V-1)  iterations,  we  can't  update  d[],  no  matter  how  many 
iterations  we  do.  Or  can  we? 

If  there  is  a  negative  cycle  in  a  graph,  even  after  (V-1)  iterations,  we  can  update  d[].  This  happens  because  for  every 
iteration,  traversing  through  the  negative  cycle  always  decreases  the  cost  of  the  shortest  path.  This  is  why  Bellman- 
Ford  algorithm  limits  the  number  of  iterations  to  (V-1).  If  we  used  Dijkstra's  Algorithm  here,  we'd  be  stuck  in  an 
endless  loop.  Flowever,  let's  concentrate  on  finding  negative  cycle. 

Let's  assume,  we  have  a  graph: 


Let's  pick  vertex  1  as  the  source.  After  applying  Bellman-Ford's  single  source  shortest  path  algorithm  to  the  graph, 
we'll  find  out  the  distances  from  the  source  to  all  the  other  vertices. 


d[3]=2  d[4]=4 

This  is  how  the  graph  looks  like  after  (V-1)  =  3  iterations.  It  should  be  the  result  since  there  are  4  edges,  we  need  at 
most  3  iterations  to  find  out  the  shortest  path.  So  either  this  is  the  answer,  or  there  is  a  negative  weight  cycle  in  the 
graph.  To  find  that,  after  (V-1)  iterations,  we  do  one  more  final  iteration  and  if  the  distance  continues  to  decrease,  it 
means  that  there  is  definitely  a  negative  weight  cycle  in  the  graph. 
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For  this  example:  if  we  check  2-3,  d[2]  +  cost[2][3]  will  give  us  1  which  is  less  than  d[3],  So  we  can  conclude  that 
there  is  a  negative  cycle  in  our  graph. 

So  how  do  we  find  out  the  negative  cycle?  We  do  a  bit  modification  to  Bellman-Ford  procedure: 

Procedure  NegativeCycleDetector(Graph,  source): 
n  :=  number  of  vertices  in  Graph 
for  i  f  rom  1  to  n 

d[i  :=  infinity 
end  for 

d[source]  :=  0 
for  i  f  rom  1  to  n-1 
flag  :=  false 

for  all  edges  from  (u,v)  in  Graph 
if  d[u]  +  cost[u][v]  <  d[v] 
d[v]  :=  d[u]  +  cost[u][v] 

flag  :=  true 
end  if 
end  for 

if  flag  ==  false 
break 

end  for 

for  all  edges  from  (u,v)  in  Graph 
if  d[u]  +  cost[u][v]  <  d[v] 

Return  "Negative  Cycle  Detected" 
end  if 

end  for 

Return  "No  Negative  Cycle" 

This  is  how  we  find  out  if  there  is  a  negative  cycle  in  a  graph.  We  can  also  modify  Bellman-Ford  Algorithm  to  keep 
track  of  negative  cycles. 

Section  13.3:  Why  do  we  need  to  relax  all  the  edges  at  most 
(V-1)  times 

To  understand  this  example,  it  is  recommended  to  have  a  brief  idea  on  Bellman-Ford  single  source  shortest  path  algorithm 
which  can  be  found  here 

In  Bellman-Ford  algorithm,  to  find  out  the  shortest  path,  we  need  to  relax  all  the  edges  of  the  graph.  This  process  is 
repeated  at  most  (V-1)  times,  where  V  is  the  number  of  vertices  in  the  graph. 

The  number  of  iterations  needed  to  find  out  the  shortest  path  from  source  to  all  other  vertices  depends  on  the 
order  that  we  select  to  relax  the  edges. 

Let's  take  a  look  at  an  example: 


d[l]=0  d[2]=inf  d[3]=Lnf  d[4]=inf 


Flere,  the  source  vertex  is  1 .  We  will  find  out  the  shortest  distance  between  the  source  and  all  the  other  vertices. 
We  can  clearly  see  that,  to  reach  vertex  4,  in  the  worst  case,  it'll  take  (V-1)  edges.  Now  depending  on  the  order  in 
which  the  edges  are  discovered,  it  might  take  (V-1)  times  to  discover  vertex  4.  Didn't  get  it?  Let's  use  Bellman-Ford 
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algorithm  to  find  out  the  shortest  path  here: 
We're  going  to  use  this  sequence: 


Serial 

-  + - 

1  1 

-  +  - 

i 

2 

-  +  - 

1 

3 

Edge 

|  3->4 

-  +  - . 

i 

2  ->3 

1 

l->2 

For  our  first  iteration: 

1 .  d[3]  +  cost[3][4]  =  infinity.  It  won't  change  anything. 

2.  d[2]  +  cost[2][3]  =  infinity.  It  won't  change  anything. 

3.  d[1]  +  cost[1][2]  =  2  <  d[2],  d[2]  =  2.  parent[2]  =  1. 

We  can  see  that  our  relaxation  process  only  changed  d[2].  Our  graph  will  look  like: 


d[l]=0  d[2]=2  d[3]=mf  d[4]=inf 


Second  iteration: 

1 .  d[3]  +  cost[3][4]  =  infinity.  It  won't  change  anything. 

2.  d[2]  +  cost[2][3]  =  5  <  d[3],  d[3]  =  5.  parent[3]  =  2. 

3.  It  won't  be  changed. 

This  time  the  relaxation  process  changed  d[3].  Our  graph  will  look  like: 


d[l]=0  d[2]=2  d[3]=5  d[4]=inf 


Third  iteration: 

1 .  d[3]  +  cost[3][4]  =  7  <  d[4],  d[4]  =  7.  parent[4]  =  3. 

2.  It  won't  be  changed. 

3.  It  won't  be  changed. 

Our  third  iteration  finally  found  out  the  shortest  path  to  4  from  1.  Our  graph  will  look  like: 


d[l]=0  d[2]=2  d[3]=5  d[4]=7 

So,  it  took  3  iterations  to  find  out  the  shortest  path.  After  this  one,  no  matter  how  many  times  we  relax  the  edges, 
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the  values  in  d[]  will  remain  the  same.  Now,  if  we  considered  another  sequence: 


Serial 

-  +  - 

1 

1 

-  +  - 

1 

2 

-  +  - 

1 

3 

Edge 

1 

l->2 

1 

2  ->3 

1 

3- >4 

We'd  get: 

1 .  d[1]  +  cost[1][2]  =  2  <  d[2],  d[2]  =  2. 

2.  d[2]  +  cost[2][3]  =  5  <  d[3],  d[3]  =  5. 

3.  d[3]  +  cost[3][4]  =  7  <  d[4],  d[4]  =  5. 

Our  very  first  iteration  has  found  the  shortest  path  from  source  to  all  the  other  nodes.  Another  sequence  1->2, 

3->4,  2->3  is  possible,  which  will  give  us  shortest  path  after  2  iterations.  We  can  come  to  the  decision  that,  no  matter 
how  we  arrange  the  sequence,  it  won't  take  more  than  3  iterations  to  find  out  shortest  path  from  the  source  in  this 
example. 

We  can  conclude  that,  for  the  best  case,  it'll  take  1  iteration  to  find  out  the  shortest  path  from  source.  For  the  worst 
case,  it'll  take  (V-1)  iterations,  which  is  why  we  repeat  the  process  of  relaxation  (V-1)  times. 
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Chapter  14:  Line  Algorithm 

Line  drawing  is  accomplished  by  calculating  intermediate  positions  along  the  line  path  between  two  specified 
endpoint  positions.  An  output  device  is  then  directed  to  fill  in  these  positions  between  the  endpoints. 

Section  14.1:  Bresenham  Line  Drawing  Algorithm 

Background  Theory:  Bresenham's  Line  Drawing  Algorithm  is  an  efficient  and  accurate  raster  line  generating 
algorithm  developed  by  Bresenham.  It  involves  only  integer  calculation  so  it  is  accurate  and  fast.  It  can  also  be 
extended  to  display  circles  another  curves. 

In  Bresenham  line  drawing  algorithm: 

For  Slope  |  m  |  <1 : 

Either  value  of  x  is  increased 

OR  both  x  and  y  is  increased  using  decision  parameter. 

For  Slope  |  m  |  >1 : 

Either  value  of  y  is  increased 

OR  both  x  and  y  is  increased  using  decision  parameter. 

Algorithm  for  slope  |  m  |  <1: 

1 .  Input  two  end  points  (x1,y1 )  and  (x2,y2)  of  the  line. 

2.  Plot  the  first  point  (x1,y1). 

3.  Calculate 

Delx  =  |  x2  -xl  | 

Dely  =  |  y2  —  yl  | 

4.  Obtain  the  initial  decision  parameter  as 
P  =  2  *  dely  -  delx 

5.  For  I  =  0  to  delx  in  step  of  1 

If  p  <  0  then 
XI  =  xl  +  1 
Pot(x1,y1) 

P  =  p+  2  dely 

Else 

XI  =x1  +  1 
Yl  =y1  +  7 
Plot(x1,y1) 

P  =  p  +  2dely-  2  *  delx 
End  if 

End  for 

6.  END 

Source  Code: 
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/*  A  C  program  to  implement  Bresenham  line  drawing  algorithm  for  lml<l  */ 

#include<stdio . h> 

#include<conio . h> 

#in clude< graphics. h> 

#include<math . h> 

int  main() 

{ 

int  gdriver=DETECT, gmode 

int  xl ,  yl ,  x2 , y2, delx, dely, p, i 

initgraph(&gdriver ,  &gmode,  "c  :  WTCWBGI" )  ; 

printf( "Enter  the  intial  points:  "); 
scant ( "%d " , &x1 )  ; 
scant ( "%d " , &y1 )  ; 

printf( "Enter  the  end  points:  "); 
scant ( "%d " , &x2)  ; 
scant ( "%d " , &y2)  ; 

putpixel(x1 , yl , RED) ; 

delx=fabs (x2-x1 )  ; 
dely=fabs (y2  yl )  ; 
p=(2*dely)-delx ; 
for(i=0;i<delx; i++) { 

if (p<0) 

{ 

xl  =x1 +1  ; 

putpixel(x1 , yl , RED) ; 

p=p+(2*dely)  ; 

} 

else 

{ 

xl  =x1 +1  ; 
yl  =y1+1  ; 

putpixel(x1 , yl , RED) ; 
p=p+(2*dely) - (2*delx)  ; 

} 

} 

getch( )  ; 
closegraph( )  ; 

return  0; 

} 

Algorithm  for  slope  |  m  |  >1: 

1 .  Input  two  end  points  (x1,y1 )  and  (x2,y2)  of  the  line. 

2.  Plot  the  first  point  (xl.yl). 

3.  Calculate 

Delx  =  |  x2  -  xl  | 

Dely  =  |  y2  —  yl  | 

4.  Obtain  the  initial  decision  parameter  as 
P  =  2  *  delx  -  dely 

5.  For  I  =  0  to  dely  in  step  of  1 

If  p  <  0  then 
yl  =  yl  +  1 
Pot(x1  ,y1 ) 
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P  =  p+  2  delx 


Else 

XI  =xl  +  1 
Yl  =y1  +  1 
Plot(x1,y1) 

P  =  p  +  2delx  -  2  *  dely 
End  if 
End  for 

6.  END 

Source  Code: 

/*  A  C  program  to  implement  Bresenham  line  drawing  algorithm  for  lml>l  */ 

#include<stdio . h> 

#include<conio . h> 

#include< graphics . h> 

#include<math . h> 
int  main() 

{ 

int  gdriver=DETECT, gmode 

int  xl , yl , x2, y2 , delx, dely , p, i 

initgraph  (&gd  river,  &gmode,  "c :  WTCWBGI" )  ; 

printf( "Enter  the  intial  points:  "); 

scanf ( "%d " , &x1 )  ; 

scanf ( "%d " , &y1 )  ; 

printf( "Enter  the  end  points:  "); 

scanf ( "%d " , &x2)  ; 

scanf ( "%d " , &y2)  ; 

putpixel(x1 , yl , RED) ; 

delx=fabs (x2-x1 )  ; 

dely=fabs(y2  yl ) ; 

p=(2*delx)-dely 

for (i=0 ; i<delx ; i++) { 

if (p<0) 

{ 

y 1 =y1 +1  ; 

putpixel(x1 , yl , RED) ; 
p=p+(2*delx) ; 

} 

else 

{ 

xl =x1 +1  ; 
yl =y1 +1  ; 

putpixel(x1 , yl , RED) ; 
p=p+(2*delx) - (2*dely) ; 

} 

} 

getch() ; 
closegraph( ) ; 

return  0; 

} 
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Chapter  15:  Floyd-Warshall  Algorithm 

Section  15.1:  All  Pair  Shortest  Path  Algorithm 

Flovd-Warshall's  algorithm  is  for  finding  shortest  paths  in  a  weighted  graph  with  positive  or  negative  edge  weights. 
A  single  execution  of  the  algorithm  will  find  the  lengths  (summed  weights)  of  the  shortest  paths  between  all  pair  of 
vertices.  With  a  little  variation,  it  can  print  the  shortest  path  and  can  detect  negative  cycles  in  a  graph.  Floyd- 
Warshall  is  a  Dynamic-Programming  algorithm. 

Let's  look  at  an  example.  We're  going  to  apply  Floyd-Warshall's  algorithm  on  this  graph: 


First  thing  we  do  is,  we  take  two  2D  matrices.  These  are  adjacency  matrices.  The  size  of  the  matrices  is  going  to  be 
the  total  number  of  vertices.  For  our  graph,  we  will  take  4*4  matrices.  The  Distance  Matrix  is  going  to  store  the 
minimum  distance  found  so  far  between  two  vertices.  At  first,  for  the  edges,  if  there  is  an  edge  between  u-v  and  the 
distance/weight  is  w,  we'll  store:  distance  [u]  [v]  =  w.  For  all  the  edges  that  doesn't  exist,  we're  gonna  put  infinity. 
The  Path  Matrix  is  for  regenerating  minimum  distance  path  between  two  vertices.  So  initially,  if  there  is  a  path 
between  u  and  v,  we're  going  to  put  path[u]  [v]  =  u.  This  means  the  best  way  to  come  to  vertex-v  from  vertex-u 
is  to  use  the  edge  that  connects  v  with  u.  If  there  is  no  path  between  two  vertices,  we're  going  to  put  N  there 
indicating  there  is  no  path  available  now.  The  two  tables  for  our  graph  will  look  like: 
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distance  path 


Since  there  is  no  loop,  the  diagonals  are  set  N.  And  the  distance  from  the  vertex  itself  is  0. 

To  apply  Floyd-Warshall  algorithm,  we're  going  to  select  a  middle  vertex  k.  Then  for  each  vertex  i,  we're  going  to 
check  if  we  can  go  from  i  to  k  and  then  k  to  j,  where  j  is  another  vertex  and  minimize  the  cost  of  going  from  i  to  j.  If 
the  current  distance[i][j]  is  greater  than  distance[i][k]  +  distance[k][j],  we're  going  to  put  distance[i][j]  equals  to 
the  summation  of  those  two  distances.  And  the  path[i][j]  will  be  set  to  path[k][j],  as  it  is  better  to  go  from  i  to  k, 
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and  then  k  to  j.  All  the  vertices  will  be  selected  as  k.  We'll  have  3  nested  loops:  for  k  going  from  1  to  4,  i  going  from 
1  to  4  and  j  going  from  1  to  4.  We're  going  check: 

if  distance[i]  [  j ]  >  distance[i] [k  +  distance [k ][ j ] 
distance[i  [j]  :=  distance[i  [k  +  distance[k  [ j ] 
path [ i] [ j ]  :=  path [ k] [ j ] 
end  if 

So  what  we're  basically  checking  is,  for  every  pair  of  vertices,  do  we  get  a  shorter  distance  by  going  through  another 
vertex?The  total  number  of  operations  for  our  graph  will  be  4  *  4  *  4  =  64.  That  means  we're  going  to  do  this  check 
64  times.  Let's  look  at  a  few  of  them: 

When  k  =  1,  i  =  2  and  j  =  3,  distance[i][j]  is  -2,  which  is  not  greater  than  distance[i][k]  +  distance[k][j]  =  -2  +  0  =  -2. 

So  it  will  remain  unchanged.  Again,  when  k  =  1,  i  =  4  and  j  =  2,  distance[i][j]  =  infinity,  which  is  greater  than 

distance[i][k]  +  distance[k][j]  =  1  +  3  =  4.  So  we  put  distance[i][j]  =  4,  and  we  put  path[i][j]  =  path[k][j]  =  1.  What 
this  means  is,  to  go  from  vertex-4  to  vertex-2,  the  path  4->1->2  is  shorter  than  the  existing  path.  This  is  how  we 
populate  both  matrices.  The  calculation  for  each  step  is  shown  here.  After  making  necessary  changes,  our  matrices 
will  look  like: 


+ - + - + - + - + - + 

I  |  1  |  2  |  3  |  4  | 

+ - + - + - + - + - + 

|  1  |  0  |  3  |  1  |  3  | 

+ - + - + - + - + - + 

|  2  |  1  |  0  |  -2  |  0  | 

+ - + - + - + - + - + 

|  3  |  3  |  6  |  0  |  2  | 

+ - + - + - + - + - + 

|  4  |  1  |  4  |  2  |  0  | 

+ - + - + - + - + - + 

distance 


+ - + - + - + - + - + 

I  |  1  |  2  |  3  |  4  | 

+ - + - + - + - + - + 

|  1  |  N  |  1  |  2  |  3  | 

+ - + - + - + - + - + 

|  2  |  4  |  N  |  2  |  3  | 

+ - + - + - + - + - + 

|  3  |  4  |  1  |  N  |  3  | 

+ - + - + - + - + - + 

|  4  |  4  |  1  |  2  |  N  | 

+ - + - + - + - + - + 

path 


This  is  our  shortest  distance  matrix.  For  example,  the  shortest  distance  from  1  to  4  is  3  and  the  shortest  distance 
between  4  to  3  is  2.  Our  pseudo-code  will  be: 

Procedure  Floyd  Warshall(Graph) : 

for  k  f  rom  1  to  V  / /  V  denotes  the  number  of  vertex 

for  i  f  rom  1  to  V 
for  j  from  1  to  V 

if  distance[i  [j]  >  distance[i  [k  +  distance[k  [j] 
distance [i] [j ]  :=  distance[i] [k]  +  distance[k] [ j ] 
path [ i  ]  [ j  ]  :=  path [k  ]  [j  ] 
end  if 
end  for 
end  for 
end  for 

Printing  the  path: 

To  print  the  path,  we'll  check  the  Path  matrix.  To  print  the  path  from  u  to  v,  we'll  start  from  path[u][v].  We'll  set 
keep  changing  v  =  path[u][v]  until  we  find  path[u][v]  =  u  and  push  every  values  of  path[u][v]  in  a  stack.  After 
finding  u,  we'll  print  u  and  start  popping  items  from  the  stack  and  print  them.  This  works  because  the  path  matrix 
stores  the  value  of  the  vertex  which  shares  the  shortest  path  to  v  from  any  other  node.  The  pseudo-code  will  be: 

Procedure  PrintPath(source,  destination) : 
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s  =  Stack() 

S . push (destination) 

while  Path [ source | [ destination  is  not  equal  to  source 
S. push (Path [sou roe  [destination] ) 
destination  :=  Path [source] [destination 
end  while 
print  ->  source 
while  S  is  not  empty 
print  ->  S.pop 
end  while 

Finding  Negative  Edge  Cycle: 

To  find  out  if  there  is  a  negative  edge  cycle,  we'll  need  to  check  the  main  diagonal  of  distance  matrix.  If  any  value 
on  the  diagonal  is  negative,  that  means  there  is  a  negative  cycle  in  the  graph. 

Complexity: 

The  complexity  of  Floyd-Warshall  algorithm  is  0(V3)  and  the  space  complexity  is:  0(V2). 
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Chapter  16:  Catalan  Number  Algorithm 

Section  16.1:  Catalan  Number  Algorithm  Basic  Information 

Catalan  numbers  algorithm  is  Dynamic  Programming  algorithm. 

In  combinatorial  mathematics,  the  Catalan  numbers  form  a  sequence  of  natural  numbers  that  occur  in  various 
counting  problems,  often  involving  recursively-defined  objects.  The  Catalan  numbers  on  nonnegative  integers  n  are 
a  set  of  numbers  that  arise  in  tree  enumeration  problems  of  the  type,  'In  how  many  ways  can  a  regular  n-gon  be 
divided  into  n-2  triangles  if  different  orientations  are  counted  separately?1 

Application  of  Catalan  Number  Algorithm: 

1 .  The  number  of  ways  to  stack  coins  on  a  bottom  row  that  consists  of  n  consecutive  coins  in  a  plane,  such  that 
no  coins  are  allowed  to  be  put  on  the  two  sides  of  the  bottom  coins  and  every  additional  coin  must  be  above 
two  other  coins,  is  the  nth  Catalan  number. 

2.  The  number  of  ways  to  group  a  string  of  n  pairs  of  parentheses,  such  that  each  open  parenthesis  has  a 
matching  closed  parenthesis,  is  the  nth  Catalan  number. 

3.  The  number  of  ways  to  cut  an  n+2-sided  convex  polygon  in  a  plane  into  triangles  by  connecting  vertices  with 
straight,  non-intersecting  lines  is  the  nth  Catalan  number.  This  is  the  application  in  which  Euler  was 
interested. 

Using  zero-based  numbering,  the  nth  Catalan  number  is  given  directly  in  terms  of  binomial  coefficients  by  the 
following  equation. 


_  1  /2n\  _  _  tt  n  +  k 

"  ~  Sllnj  "  (n  +  l)!n!  “  HT 


for  n  >  0. 


Example  of  Catalan  Number: 

Here  value  of  n  =  4.(Best  Example  -  From  Wikipedia) 
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Auxiliary  Space:  0(n) 
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Time  Complexity:  0(nA2) 
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Chapter  17:  polunomial-time  bounded 
algorithm  for  Minimum  Vertex  Cover 

Variable  Meaning 

G  Input  connected  un-directed  graph 

X  Set  of  vertices 

C  Final  set  of  vertices 

This  is  a  polynomial  algorithm  for  getting  the  minimum  vertex  cover  of  connected  undirected  graph.  The  time 
complexity  of  this  algorithm  is  0(n2) 

Section  17.1:  Algorithm  Pseudo  Code 

Algorithm  PMinVertexCover  (graph  G) 

Input  connected  graph  G 

Output  Minimum  Vertex  Cover  Set  C 

Set  C  <-  new  Set<Vertex>( ) 

Set  X  <-  new  Set<Vertex>( ) 

X  <-  G .  getAHVerticiesArrangedDescendinglyByDegree( ) 

for  v  in  X  do 

List<Vertex>  adjacentVerticesI  <-  G.getAdjacent(v) 
if  !C  contains  any  of  adjacentVerticesI  then 
C . add (v) 

for  vertex  in  C  do 

List<vertex>  adjacentVertices2  <-  G.adjacentVertecies(vertex) 
if  C  contains  any  of  ad jacentVertices2  then 
C . remove( vertex) 


return  C 


C  is  the  minimum  vertex  cover  of  graph  G 


we  can  use  bucket  sort  for  sorting  the  vertices  according  to  its  degree  because  the  maximum  value  of 
degrees  is  (n-1 )  where  n  is  the  number  of  vertices  then  the  time  complexity  of  the  sorting  will  be  O(n) 
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Chapter  18:  Multithreaded  Algorithms 

Examples  for  some  multithreaded  algorithms. 

Section  18.1:  Square  matrix  multiplication  multithread 

multiply  square  matrix- parallel(A,  B) 
n  =  A. lines 

C  =  Matrix(n,n)  //create  a  new  matrix  n*n 

parallel  for  i  =  1  to  n 

parallel  for  j  =  1  to  n 
C[i][j]  =  0 
pour  k  =  1  to  n 

C[i][j]  =  C[i][j]  +  A[i][k]*B[k][j] 

return  C 

Section  18.2:  Multiplication  matrix  vector  multithread 


matrix  vector(A,x) 
n  =  A. lines 

y  =  Vector(n)  //create  a  new  vector  of  length  n 

parallel  for  i  =  1  to  n 

y[i]  =  0 

parallel  for  i  =  1  to  n 
for  j  =  1  to  n 

y[i]  =  y[i]  +  A [ i ] [ j ]*x[ j ] 

return  y 

Section  18.3:  merge-sort  multithread 

A  is  an  array  and  p  and  q  indexes  of  the  array  such  as  you  gonna  sort  the  sub-array  A[p..r],  B  is  a  sub-array  which  will 
be  populated  by  the  sort. 

A  call  to  p-merge-sort(A,p,r,B,s)  sorts  elements  from  A[p..r]  and  put  them  in  B[s..s+r-p], 

p  merge -sort(A,  p,  r,  B,  s) 
n  =  r  p+1 
if  n==1 

B  [  s  ]  =  A  [  p  ] 

else 

T  =  new  Array(n)  //create  a  new  array  T  of  size  n 

q  =  f loor( (p+r) /2) ) 

q_prime  =  q  p+1 

spawn  p  merge  sort(A,  p,  q,  T,  1 ) 

p- merge  sort(A, q+1 , r, T, q_prime+1 ) 

sync 

p-merge(T,  1 , q_prime, q_prime+1 ,n,B,s) 

Here  is  the  auxiliary  function  that  performs  the  merge  in  parallel. 

p-merge  assumes  that  the  two  sub-arrays  to  merge  are  in  the  same  array  but  doesn't  assume  they  are  adjacent  in 
the  array.  That's  why  we  need  p1,r1,p2,r2. 

p  merge(T, pi , rl , p2, r2 , A, p3) 
nl  =  rl  -pi  +1 
n2  =  r2  p2+1 

if  n1<n2  //check  if  nl>=n2 
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permute  pi  and  p2 
permute  rl  and  r2 
permute  nl  and  n2 

if  n1==0  //both  empty? 

return 

else 

ql  =  floor( (pl  +  rl  )/2) 

q2  =  dichotomic-search(T[q1 ] , T, p2, r2) 

q3  =  p3  +  (ql -pi )  +  (q2  p2) 

A[q3]  =  T[q1] 

spawn  p  merge(T, pi , ql -1 , p2, q2 - 1 , A, p3) 
p-merge(T,q1+1, r1,q2, r2,A,q3+1) 
sync 

And  here  is  the  auxiliary  function  dichotomic-search. 

x  is  the  key  to  look  for  in  the  sub-array  T[p..r]. 

dichotomic~search(x,T, p, r) 
inf  =  p 

sup  =  max(p, r+1 ) 
while  inf<sup 

half  =  floor( (inf+sup) /2) 
if  x<=T[half] 
sup  =  half 

else 

inf  =  half+1 

return  sup 
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Chapter  19:  Knuth  Morris  Pratt  (KMP) 
Algorithm 

The  KMP  is  a  pattern  matching  algorithm  which  searches  for  occurrences  of  a  "word"  W  within  a  main  "text  string"  S 
by  employing  the  observation  that  when  a  mismatch  occurs,  we  have  the  sufficient  information  to  determine  where 
the  next  match  could  begin.We  take  advantage  of  this  information  to  avoid  matching  the  characters  that  we  know 
will  anyway  match.The  worst  case  complexity  for  searching  a  pattern  reduces  to  O(n). 

Section  19.1:  KMP-Example 

Algorithm 

This  algorithm  is  a  two  step  process. First  we  create  a  auxiliary  array  lps[]  and  then  use  this  array  for  searching  the 
pattern. 

Preprocessing : 

1 .  We  pre-process  the  pattern  and  create  an  auxiliary  array  lps[]  which  is  used  to  skip  characters  while 
matching. 

2.  Here  lps[]  indicates  longest  proper  prefix  which  is  also  suffix.A  proper  prefix  is  prefix  in  which  whole  string  is 
not  included. For  example,  prefixes  of  string  ABC  are  “  ”,  “A”,  “AB”  and  “ABC”.  Proper  prefixes  are  “  ”,  “A”  and 
“AB”.  Suffixes  of  the  string  are  “  ”,  “C”,  “BC”  and  “ABC”. 

Searching 

1 .  We  keep  matching  characters  txt[i]  and  pat[j]  and  keep  incrementing  i  and  j  while  pat[j]  and  txt[i]  keep 
matching. 

2.  When  we  see  a  mismatch, we  know  that  characters  pat[0..j-1]  match  with  txt[i-j+1  ...i-1].We  also  know  that 
lps[j-1]  is  count  of  characters  of  pat[0...j-1]  that  are  both  proper  prefix  and  suffix.From  this  we  can  conclude 
that  we  do  not  need  to  match  these  IpsU-l]  characters  with  txt[i-j...i-1]  because  we  know  that  these 
characters  will  match  anyway. 


Implementaion  in  Java 

public  class  KMP  { 

public  static  void  main(String[ ]  args)  { 

//  TODO  Auto-generated  method  stub 
String  str  =  "abcabdabc"; 

String  pattern  =  "abc"; 

KMP  obj  =  new  KMP( )  ; 

System. out .println(obj.patternExistKMP(str. toCharAr ray( ) ,  pattern . toCharAr ray ())) ; 


public  int [ ]  computeLPS(char [ ]  str){ 
int  lps[]  =  new  int[str. length] ; 

lps [ 0 ]  =  0; 
int  j  =  0 ; 

for(int  i  =1 ; i<str . length ; i++) { 
if (str [ j ]  ==  str [i] ) { 
lps [ i ]  =  j+1 ; 
j++; 
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} 


i++; 

}else{ 

if ( j !=0){ 

j  =  lps[ j-1 ] ; 

}else{ 

lps [ i ]  =  j+1 ; 
i++  ; 

} 

} 


return  lps 

} 

public  boolean  patternExistKMP(char [  ]  text,char[  pat){ 
int[]  lps  =  computeLPS(pat) ; 
int  i=0,j=0; 

while(i<text . length  &&  j<pat . length) { 
if ( text [i]  ==  pat [ j  ] )  { 
i++  ; 

j++; 

}else{ 

if( j !=0){ 

j  =  lps[ j-1  ]  ; 

}else{ 

i++  ; 

} 

} 

} 

if ( j==pat . length) 

return  true 
return  false 


} 
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Chapter  20:  Edit  Distance  Dynamic 
Algorithm 

Section  20.1:  Minimum  Edits  required  to  convert  string  1  to 
string  2 

The  problem  statement  is  like  if  we  are  given  two  string  strl  and  str2  then  how  many  minimum  number  of 
operations  can  be  performed  on  the  strl  that  it  gets  converted  to  str2.The  Operations  can  be: 

1.  Insert 

2.  Remove 

3.  Replace 

For  Example 

Input:  strl  =  "geek",  str2  =  "gesek" 

Output :  1 

We  only  need  to  insert  s  in  first  string 

Input:  strl  =  "march",  str2  =  "cart" 

Output :  3 

We  need  to  replace  m  with  c  and  remove  character  c  and  then  replace  h  with  t 

To  solve  this  problem  we  will  use  a  2D  array  dp[n+1][m+1]  where  n  is  the  length  of  the  first  string  and  m  is  the 
length  of  the  second  string.  For  our  example,  if  strl  is  azcef  and  str2  is  abcdef  then  our  array  will  be  dp[6][7]and 
our  final  answer  will  be  stored  at  dp[5][6]. 


(a)  (b)  (c)  (d)  (e)  (f) 
+  — + — + —  +  —  +  — + —  +  —  + 

|  0  |  1  |  2  |  3  |  4  |  5  |  6  | 

+ —  + —  + —  + —  + —  + —  + —  + 

(a) 

+ — + — + —  + — + — + —  + — + 

(z) 

+  -  - 
(c) 

+  -  - 

(e) 

+  — + — + — + — +  — + —  + — + 

(f) 

+  — + — + — + — +  — + —  + — + 


For  dp[1][1]  we  have  to  check  what  can  we  do  to  convert  a  into  a. It  will  be  O.For  dp[1][2]  we  have  to  check  what  can 
we  do  to  convert  a  into  ab.lt  will  be  1  because  we  have  to  insert  b.So  after  1  st  iteration  our  array  will  look  like 

(a)  (b)  (c)  (d)  (e)  (f) 

+  —  +  — + —  +  —  +  — + —  +  —  + 

|  0  |  1  |  2  |  3  |  4  |  5  |  6  | 

+  —  +  —  +  —  +  —  +  —  +  —  +  —  + 

(a) |  1  |  0  |  1  |  2  |  3  |  4  |  5  | 

+  —  +  —  +  —  +  —  +  —  +  —  +  —  + 

(z)l  2  |  |  |  |  |  |  | 

+  —  +  —  +  —  +  —  +  —  +  —  +  —  + 

(c) |  3  |  |  |  |  |  |  | 
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+ —  +  —  + —  +  —  +  —  +  —  +  —  + 

(e)  |  4  |  |  |  |  |  |  | 

+ - + - + - + - + - + - + - + 

(f) l  5  |  |  |  |  |  |  | 

+  —  +  —  +  —  +  —  + —  +  —  +  —  + 

For  iteration  2 

For  dp[2][1]  we  have  to  check  that  to  convert  az  to  a  we  need  to  remove  z,  hence  dp[2][1]  will  be  I.Similary  for 
dp[2][2]  we  need  to  replace  z  with  b,  hence  dp[2][2]  will  be  I.So  after  2nd  iteration  our  dp[]  array  will  look  like. 

(a)  (b)  (c)  (d)  (e)  (f) 

+ — + — + — + — + — + —  + — + 

|  0  |  1  |  2  |  3  |  4  |  5  |  6  | 

+  —  +  —  +  —  +  —  +  —  +  —  +  —  + 

(a) |  1  |  0  |  1  |  2  |  3  |  4  |  5  | 

+  —  +  —  +  —  +  —  +  —  +  —  +  —  + 

(z)l  2  |  1  |  1  |  2  |  3  |  4  |  5  | 

+ —  + —  +  —  + —  + —  + —  + —  + 

(c) |  3  |  |  |  |  |  |  | 

+ —  + —  + —  + —  + —  + —  + —  + 

(e)  |  4  |  |  |  |  |  |  | 

+  —  +  — + —  +  —  +  — + —  +  —  + 

(f) l  5  |  |  |  |  |  |  | 

+  —  +  —  +  —  +  —  +  —  +  —  +  —  + 

So  our  formula  will  look  like 

if  characters  are  same 

dp [ i ] [ j  ]  =  dp [ i-1  ]  [ j-1  ] ; 

else 

dp[i] [ j ]  =  1  +  Min(dp [i-1  ]  [  j ] ,  dp[i][j-1],  dp [ i-1 ] [ j-1 ] ) 

After  last  iteration  our  dp[]  array  will  look  like 

(a)  (b)  (c)  (d)  (e)  (f) 

+ — + — + —  +  —  +  — + —  + —  + 

|  0  |  1  |  2  |  3  |  4  |  5  |  6  | 

+ —  + —  + —  +  —  + —  + —  + —  + 

(a) |  1  |  0  |  1  |  2  |  3  |  4  |  5  | 

+  —  +  —  +  —  +  —  + —  +  —  +  —  + 

(z)l  2  |  1  |  1  |  2  |  3  |  4  |  5  | 

+  —  +  —  +  —  +  —  +  —  +  —  +  —  + 

(c) |  3  |  2  |  2  |  1  |  2  |  3  |  4  | 

+ —  + —  + —  + —  +  —  +  —  +  —  + 

(e) |  4  |  3  |  3  |  2  |  2  |  2  |  3  | 

+  —  +  —  +  —  +  —  +  —  +  —  +  —  + 

(f )  I  5  |  4  |  4  |  2  |  3  |  3  |  3  | 

+  —  +  —  +  —  +  —  +  —  +  —  +  —  + 

Implementation  in  Java 

public  int  getMinConversions(String  strl,  String  str2){ 

int  dp[][]  =  new  int [ strl . lengthf )+1 ] [str2 . length( )+1 ] ; 
for(int  i=0 ; i<=st rl . length( )  i++){ 

for  (int  j=0 ;  j<=str2  .  length  ( ) ,']++){ 
if ( i==0) 
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dp [ i ] [ j ]  =  j ; 
else  if(j==0) 

dp [ i ] [ j ]  =  i 

else  if (strl .charAt(i  1)  ==  str2.charAt( j-1 )) 
dp[i] [j ]  =  dp [ i-1 ] [j-1 ] ; 

else{ 

dp[i][j]  =  1  +  Math  .min(dp[ i-1 ][ j ] ,  Math. min(dp[i] [j-1 ] ,  dp[i  1][j-1])); 

} 

} 

} 

return  dp [strl . length ( ) ] [ st r2 . length ( ) ] ; 


Time  Complexity 

0(nA2) 
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Chapter  21:  Online  algorithms 

Theory 

Definition  1:  An  optimization  problem  n  consists  of  a  set  of  instances  in.  For  every  instance  oein  there  is  a  set 
Zo  of  solutions  and  a  objective  function  fa  :  Za  -*•  'JteO  which  assigns  apositive  real  value  to  every  solution. 

We  say  OPT(a)  is  the  value  of  an  optimal  solution,  A(a)  is  the  solution  of  an  Algorithm  A  for  the  problem  n  and 
wA(a)=fa(A(a))  its  value. 

Definition  2:  An  online  algorithm  A  for  a  minimization  problem  n  has  a  competetive  ratio  of  r  >  1  if  there  is  a 
constant  TesJt  with 


wA(a)  =  fa(A(a))  2  r  •  OPT(&sigma)  +  t 


for  all  instances  aelfl.  A  is  called  a  r-competitive  online  algorithm.  Is  even 


wA(ct)  <  r  •  OPT(&sigma) 


for  all  instances  aein  then  A  is  called  a  strictly  r-competitive  online  algorithm. 

Proposition  1.3:  LRU  and  FWF  are  marking  algorithm. 

Proof:  At  the  beginning  of  each  phase  (except  for  the  first  one)  FWF  has  a  cache  miss  and  cleared  the  cache,  that 
means  we  have  k  empty  pages.  In  every  phase  are  maximal  k  different  pages  requested,  so  there  will  be  now 
eviction  during  the  phase.  So  FWF  is  a  marking  algorithm. 

Lets  assume  LRU  is  not  a  marking  algorithm.  Then  there  is  an  instance  a  where  LRU  a  marked  page  x  in  phase  i 
evicted.  Let  at  the  request  in  phase  i  where  x  is  evicted.  Since  x  is  marked  there  has  to  be  a  earlier  request  at*  for  x 
in  the  same  phase,  so  t*  <  t.  After  t*  x  is  the  caches  newest  page,  so  to  got  evicted  at  t  the  sequence  at*+1  ,...,at  has 
to  request  at  least  k  from  x  different  pages.  That  implies  the  phase  i  has  requested  at  least  k+1  different  pages 
which  is  a  contradictory  to  the  phase  definition.  So  LRU  has  to  be  a  marking  algorithm. 

Proposition  1.4:  Every  marking  algorithm  is  strictly  k-competitive. 

Proof:  Let  a  be  an  instance  for  the  paging  problem  and  I  the  number  of  phases  for  a.  Is  I  =  1  then  is  every  marking 
algorithm  optimal  and  the  optimal  offline  algorithm  cannot  be  better. 

We  assume  I  >  2.  the  cost  of  every  marking  algorithm  for  instance  a  is  bounded  from  above  with  I  •  k  because  in 
every  phase  a  marking  algorithm  cannot  evict  more  than  k  pages  without  evicting  one  marked  page. 

Now  we  try  to  show  that  the  optimal  offline  algorithm  evicts  at  least  k+l-2  pages  for  a,  k  in  the  first  phase  and  at 
least  one  for  every  following  phase  except  for  the  last  one.  For  proof  lets  define  1-2  disjunct  subsequences  of  a. 
Subsequence  i  e  {1  ,...,1-2}  starts  at  the  second  position  of  phase  i+1  and  end  with  the  first  position  of  phase  i+2. 

Let  x  be  the  first  page  of  phase  i+1 .  At  the  beginning  of  subsequence  i  there  is  page  x  and  at  most  k-1  different 
pages  in  the  optimal  offline  algorithms  cache.  In  subsequence  i  are  k  page  request  different  from  x,  so  the  optimal 
offline  algorithm  has  to  evict  at  least  one  page  for  every  subsequence.  Since  at  phase  1  beginning  the  cache  is  still 
empty,  the  optimal  offline  algorithm  causes  k  evictions  during  the  first  phase.  That  shows  that 

wA(a)  <  l-k  <  (k+l-2)k  <  OPT(a)  •  k 

Corollary  1.5:  LRU  and  FWF  are  strictly  k-competitive. 
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Is  there  no  constant  r  for  which  an  online  algorithm  A  is  r-competitive,  we  call  A  not  competitive. 

Proposition  1.6:  LFU  and  LIFO  are  not  competitive. 

Proof:  Let  I  >  2  a  constant,  k  >  2  the  cache  size.  The  different  cache  pages  are  nubered  1  ,...,k+1 .  We  look  at  the 
following  sequence: 

(t  —  (1(,  2(, . . . ,  (k  —  1);,  (k.  k  +  1),_1) 

First  page  1  is  requested  I  times  than  page  2  and  so  one.  At  the  end  there  are  (1-1 )  alternating  requests  for  page  k 
and  k+1 . 

LFU  and  LIFO  fill  their  cache  with  pages  1-k.  When  page  k+1  is  requested  page  k  is  evicted  and  vice  versa.  That 
means  every  request  of  subsequence  (k,k+1  )l-1  evicts  one  page.  In  addition  their  are  k-1  cache  misses  for  the  first 
time  use  of  pages  1  -(k-1 ).  So  LFU  and  LIFO  evict  exact  k-1  +2(1-1 )  pages. 

Now  we  must  show  that  for  every  constant  Te'J?  and  every  constan  r  <  1  there  exists  an  I  so  that 

«-’LFu(ct)  =  wlifo(o-)  >  r  ■  OPT(a)  +  r 

which  is  equal  to 

k  -  l  +  2(1  -  l)  >  r(k  + 1)  +  t  /  >  l  +  - '  ^ 

To  satisfy  this  inequality  you  just  have  to  choose  I  sufficient  big.  So  LFU  and  LIFO  are  not  competetive. 

Proposition  1.7:  There  is  no  r-competetive  deterministic  online  algorithm  for  paging  with  r  <  k. 

Sources 
Basic  Material 

1 .  Script  Online  Algorithms  (german),  Heiko  Roeglin,  University  Bonn 

2.  Page  replacement  algorithm 

Further  Reading 

1 .  Online  Computation  and  Competetive  Analysis  by  Allan  Borodin  and  Ran  El-Yaniv 

Source  Code 

1.  Source  code  for  offline  caching 

2.  Source  code  for  adversary  game 

Section  21.1:  Paging  (Online  Caching) 

Preface 

Instead  of  starting  with  a  formal  definition,  the  goal  is  to  approach  these  topic  via  a  row  of  examples,  introducing 
definitions  along  the  way.  The  remark  section  Theory  will  consist  of  all  definitions,  theorems  and  propositions  to 
give  you  all  information  to  faster  look  up  specific  aspects. 
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The  remark  section  sources  consists  of  the  basis  material  used  for  this  topic  and  additional  information  for  further 
reading.  In  addition  you  will  find  the  full  source  codes  for  the  examples  there.  Please  pay  attention  that  to  make  the 
source  code  for  the  examples  more  readable  and  shorter  it  refrains  from  things  like  error  handling  etc.  It  also 
passes  on  some  specific  language  features  which  would  obscure  the  clarity  of  the  example  like  extensive  use  of 
advanced  libraries  etc. 


Paging 

The  paging  problem  arises  from  the  limitation  of  finite  space.  Let's  assume  our  cache  C  has  k  pages.  Now  we  want 
to  process  a  sequence  of  m  page  requests  which  must  have  been  placed  in  the  cache  before  they  are  processed.  Of 
course  if  m<=k  then  we  just  put  all  elements  in  the  cache  and  it  will  work,  but  usually  is  m>>k. 

We  say  a  request  is  a  cache  hit,  when  the  page  is  already  in  cache,  otherwise,  its  called  a  cache  miss.  In  that  case, 
we  must  bring  the  requested  page  into  the  cache  and  evict  another,  assuming  the  cache  is  full.  The  Goal  is  an 
eviction  schedule  that  minimizes  the  number  of  evictions. 

There  are  numerous  strategies  for  this  problem,  let's  look  at  some: 

1 .  First  in,  first  out  (FIFO):  The  oldest  page  gets  evicted 

2.  Last  in,  first  out  (LIFO):  The  newest  page  gets  evicted 

3.  Least  recently  used  (LRU):  Evict  page  whose  most  recent  access  was  earliest 

4.  Least  frequently  used  (LFU):  Evict  page  that  was  least  frequently  requested 

5.  Longest  forward  distance  (LFD):  Evict  page  in  the  cache  that  is  not  requested  until  farthest  in  the  future. 

6.  Flush  when  full  (FWF):  clear  the  cache  complete  as  soon  as  a  cache  miss  happened 

There  are  two  ways  to  approach  this  problem: 

1 .  offline:  the  sequence  of  page  requests  is  known  ahead  of  time 

2.  online:  the  sequence  of  page  requests  is  not  known  ahead  of  time 

Offline  Approach 

For  the  first  approach  look  at  the  topic  Applications  of  Greedy  technique.  It's  third  Example  Offline  Caching 
considers  the  first  five  strategies  from  above  and  gives  you  a  good  entry  point  for  the  following. 

The  example  program  was  extended  with  the  FWF  strategy: 

class  FWF  public  Strategy  { 

public 

FWF ( )  :  Strategy("FWF") 

{ 

} 

int  apply(int  requestlndex)  override 

{ 

for (int  i=0;  i<cacheSize  ++i) 

{ 

if(cache[i  ==  request [ requestlndex ] ) 

return  i 

//  after  first  empty  page  all  others  have  to  be  empty 

else  if(cache[i]  ==  emptyPage) 

return  i 

} 

//  no  free  pages 
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} 


return  0; 


void  update(int  cachePos,  int  requestlndex,  bool  cacheMiss)  override 

{ 


}  ; 


//  no  pages  free  ->  miss  ->  clear  cache 
if(cacheMiss  &&  cachePos  ==  0) 

{ 

for(int  i  =  1 ;  i  <  cacheSize  ++i) 
cache [i]  =  emptyPage 

} 


The  full  sourcecode  is  available  here.  If  we  reuse  the  example  from  the  topic,  we  get  the  following  output: 


Strategy  FWF 

Cache  initial:  (a,b,c) 


Request 

cache 

0  cache 

cache  2 

cache  miss 

a 

a 

b 

c 

a 

a 

b 

c 

d 

d 

X 

X 

X 

e 

d 

e 

X 

b 

d 

e 

b 

b 

d 

e 

b 

a 

a 

X 

X 

X 

c 

a 

c 

X 

f 

a 

c 

f 

d 

d 

X 

X 

X 

e 

d 

e 

X 

a 

d 

e 

a 

f 

f 

X 

X 

X 

b 

f 

b 

X 

e 

f 

b 

e 

c 

c 

X 

X 

X 

Total  cache  misses:  5 

Even  though  LFD  is  optimal,  FWF  has  fewer  cache  misses.  But  the  main  goal  was  to  minimize  the  number  of 
evictions  and  for  FWF  five  misses  mean  1 5  evictions,  which  makes  it  the  poorest  choice  for  this  example. 


Online  Approach 

Now  we  want  to  approach  the  online  problem  of  paging.  But  first  we  need  an  understanding  how  to  do  it. 
Obviously  an  online  algorithm  cannot  be  better  than  the  optimal  offline  algorithm.  But  how  much  worse  it  is?  We 
need  formal  definitions  to  answer  that  question: 

Definition  1.1:  An  optimization  problem  n  consists  of  a  set  of  instances  in.  For  every  instance  oein  there  is  a 
set  Zct  of  solutions  and  a  objective  function  fa  :  Za  ->  SJ?>0  which  assigns  apositive  real  value  to  every  solution. 
We  say  OPT(a)  is  the  value  of  an  optimal  solution,  A(a)  is  the  solution  of  an  Algorithm  A  for  the  problem  fl  and 
wA(a)=fa(A(a))  its  value. 

Definition  1.2:  An  online  algorithm  A  for  a  minimization  problem  n  has  a  competetive  ratio  of  r  >  1  if  there  is  a 
constant  Teslt  with 
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wA(ct)  =  fa(A(a))  <  r  •  OPT(a)  +  t 


for  all  instances  aein.  A  is  called  a  r-competitive  online  algorithm.  Is  even 

wA(ct)  <  r  •  OPT(o) 


for  all  instances  aein  then  A  is  called  a  strictly  r-competitive  online  algorithm. 

So  the  question  is  how  competitive  is  our  online  algorithm  compared  to  an  optimal  offline  algorithm.  In  their 
famous  book  Allan  Borodin  and  Ran  El-Yaniv  used  another  scenario  to  describe  the  online  paging  situation: 

There  is  an  evil  adversary  who  knows  your  algorithm  and  the  optimal  offline  algorithm.  In  every  step,  he  tries  to 
request  a  page  which  is  worst  for  you  and  simultaneously  best  for  the  offline  algorithm,  the  competitive  factor  of 
your  algorithm  is  the  factor  on  how  badly  your  algorithm  did  against  the  adversary's  optimal  offline  algorithm.  If 
you  want  to  try  to  be  the  adversary,  you  can  try  the  Adversary  Game  (try  to  beat  the  paging  strategies). 

Marking  Algorithms 

Instead  of  analysing  every  algorithm  separately,  let's  look  at  a  special  online  algorithm  family  for  the  paging 
problem  called  marking  algorithms. 


Let  a=(a1  ,...,ap)  an  instance  for  our  problem  and  k  our  cache  size,  than  a  can  be  divided  into  phases: 


•  Phase  1  is  the  maximal  subsequence  of  a  from  the  start  till  maximal  k  different  pages  are  requested 

•  Phase  i  >  2  is  the  maximal  subsequence  of  a  from  the  end  of  pase  i-1  till  maximal  k  different  pages  are 
requested 


For  example  with  k  =  3: 


phase  1 


ph  ase  2  ph  ase  3  phase  4' 


"S 


<7  —  a,  b.  cL  a,  e,  a,  /.  a,  /,  6,  <7.  a ,  c,  c.  d 


A  marking  algorithm  (implicitly  or  explicitly)  maintains  whether  a  page  is  marked  or  not.  At  the  beginning  of  each 
phase  are  all  pages  unmarked.  Is  a  page  requested  during  a  phase  it  gets  marked.  An  algorithm  is  a  marking 
algorithm  iff  it  never  evicts  a  marked  page  from  cache.  That  means  pages  which  are  used  during  a  phase  will  not  be 
evicted. 

Proposition  1.3:  LRU  and  FWF  are  marking  algorithm. 

Proof:  At  the  beginning  of  each  phase  (except  for  the  first  one)  FWF  has  a  cache  miss  and  cleared  the  cache,  that 
means  we  have  k  empty  pages.  In  every  phase  are  maximal  k  different  pages  requested,  so  there  will  be  now 
eviction  during  the  phase.  So  FWF  is  a  marking  algorithm. 

Let's  assume  LRU  is  not  a  marking  algorithm.  Then  there  is  an  instance  a  where  LRU  a  marked  page  x  in  phase  i 
evicted.  Let  at  the  request  in  phase  i  where  x  is  evicted.  Since  x  is  marked  there  has  to  be  a  earlier  request  at*  for  x 
in  the  same  phase,  so  t*  <  t.  After  t*  x  is  the  caches  newest  page,  so  to  got  evicted  at  t  the  sequence  at*+1  ,...,at  has 
to  request  at  least  k  from  x  different  pages.  That  implies  the  phase  i  has  requested  at  least  k+1  different  pages 
which  is  a  contradictory  to  the  phase  definition.  So  LRU  has  to  be  a  marking  algorithm. 
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Proposition  1.4:  Every  marking  algorithm  is  strictly  k-competitive. 

Proof:  Let  a  be  an  instance  for  the  paging  problem  and  I  the  number  of  phases  for  a.  Is  I  =  1  then  is  every  marking 
algorithm  optimal  and  the  optimal  offline  algorithm  cannot  be  better. 

We  assume  I  >  2.  the  cost  of  every  marking  algorithm,  for  instance,  a  is  bounded  from  above  with  I  •  k  because  in 
every  phase  a  marking  algorithm  cannot  evict  more  than  k  pages  without  evicting  one  marked  page. 

Now  we  try  to  show  that  the  optimal  offline  algorithm  evicts  at  least  k+l-2  pages  for  a,  k  in  the  first  phase  and  at 
least  one  for  every  following  phase  except  for  the  last  one.  For  proof  lets  define  1-2  disjunct  subsequences  of  a. 
Subsequence  i  e  {1  ,...,1-2}  starts  at  the  second  position  of  phase  i+1  and  end  with  the  first  position  of  phase  i+2. 

Let  x  be  the  first  page  of  phase  i+1 .  At  the  beginning  of  subsequence  i  there  is  page  x  and  at  most  k-1  different 
pages  in  the  optimal  offline  algorithms  cache.  In  subsequence  i  are  k  page  request  different  from  x,  so  the  optimal 
offline  algorithm  has  to  evict  at  least  one  page  for  every  subsequence.  Since  at  phase  1  beginning  the  cache  is  still 
empty,  the  optimal  offline  algorithm  causes  k  evictions  during  the  first  phase.  That  shows  that 

wA(ct)  <  l-k  <  (k+l-2)k  <  OPT(ct)  •  k 

Corollary  1.5:  LRU  and  FWF  are  strictly  k-competitive. 

Excercise:  Show  that  FIFO  is  no  marking  algorithm,  but  strictly  k-competitive. 

Is  there  no  constant  r  for  which  an  online  algorithm  A  is  r-competitive,  we  call  A  not  competitive 
Proposition  1.6:  LFU  and  LIFO  are  not  competitive. 

Proof:  Let  I  >  2  a  constant,  k  >  2  the  cache  size.  The  different  cache  pages  are  nubered  1  ,...,k+1 .  We  look  at  the 
following  sequence: 

c  —  (1(,  2l, . . . ,  (k  —  1);,  (k.  k  +  1),_1) 

The  first  page  1  is  requested  I  times  than  page  2  and  so  one.  At  the  end,  there  are  (1-1 )  alternating  requests  for  page 
k  and  k+1 . 

LFU  and  LIFO  fill  their  cache  with  pages  1-k.  When  page  k+1  is  requested  page  k  is  evicted  and  vice  versa.  That 
means  every  request  of  subsequence  (k,k+1  )l-1  evicts  one  page.  In  addition,  their  are  k-1  cache  misses  for  the  first 
time  use  of  pages  1  -(k-1 ).  So  LFU  and  LIFO  evict  exact  k-1  +2(1-1 )  pages. 

Now  we  must  show  that  for  every  constant  TesJ?  and  every  constant  r  <  1  there  exists  an  I  so  that 

■^LFu(o-)  =  W'LIFo(o-)  >  r  ■  OPT(cr)  +  T 

which  is  equal  to 

fc  —  1  +  2(i  —  1)  >  r(k  +  1)  +  r  <=>  <>i+'-^  +  1)  +  t-A-  +  1 

To  satisfy  this  inequality  you  just  have  to  choose  I  sufficient  big.  So  LFU  and  LIFO  are  not  competitive. 

Proposition  1.7:  There  is  no  r-competetive  deterministic  online  algorithm  for  paging  with  r  <  k. 
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The  proof  for  this  last  proposition  is  rather  long  and  based  of  the  statement  that  LFD  is  an  optimal  offline 
algorithm.  The  interested  reader  can  look  it  up  in  the  book  of  Borodin  and  El-Yaniv  (see  sources  below). 

The  Question  is  whether  we  could  do  better.  For  that,  we  have  to  leave  the  deterministic  approach  behind  us  and 
start  to  randomize  our  algorithm.  Clearly,  its  much  harder  for  the  adversary  to  punish  your  algorithm  if  it's 
randomized. 

Randomized  paging  will  be  discussed  in  one  of  next  examples... 
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Chapter  22:  Big-0  Notation 

Definition 

The  Big-0  notation  is  at  its  heart  a  mathematical  notation,  used  to  compare  the  rate  of  convergence  of  functions. 

Let  n  ->  f(n)  and  n  ->  g(n)  be  functions  defined  over  the  natural  numbers.  Then  we  say  that  f  =  0(g)  if  and 
only  if  f  (n)  /g(n)  is  bounded  when  n  approaches  infinity.  In  other  words,  f  =  0(g)  if  and  only  if  there  exists  a 
constant  A,  such  thatfor  all  n,  f(n)/g(n)  <=  A. 

Actually  the  scope  of  the  Big-0  notation  is  a  bit  wider  in  mathematics  but  for  simplicity  I  have  narrowed  it  to  what  is 
used  in  algorithm  complexity  analysis  :  functions  defined  on  the  naturals,  that  have  non-zero  values,  and  the  case 
of  n  growing  to  infinity. 

What  does  it  mean  ? 

Let's  take  the  case  of  f(n)  =  100nA2  +  10n  +  landg(n)  =  nA2.  It  is  quite  clear  that  both  of  these  functions  tend 
to  infinity  as  n  tends  to  infinity.  But  sometimes  knowing  the  limit  is  not  enough,  and  we  also  want  to  know  the  speed 
at  which  the  functions  approach  their  limit.  Notions  like  Big-0  help  compare  and  classify  functions  by  their  speed  of 
convergence. 

Let's  find  out  if  f  =  0(g)  by  applying  the  definition.  We  have  f(n) /g(n)  =  100  +  10/n  +  1 /nA2.  Since  1 0/n  is  1 0 
when  n  is  1  and  is  decreasing,  and  since  1 /nA2  is  1  when  n  is  1  and  is  also  decreasing,  we  have  f(n)/g(n)  <=  100  + 
10  +  1  =  1 1 1 .  The  definition  is  satisfied  because  we  have  found  a  bound  of  f(n) /g(n)  (1 1 1 )  and  so  f  =  0(g)  (we 
say  that  f  is  a  Big-0  of  nA2). 

This  means  that  f  tends  to  infinity  at  approximately  the  same  speed  as  g.  Now  this  may  seem  like  a  strange  thing  to 
say,  because  what  we  have  found  is  that  f  is  at  most  1 1 1  times  bigger  than  g,  or  in  other  words  when  g  grows  by  1 ,  f 
grows  by  at  most  1 1 1 .  It  may  seem  that  growing  1 1 1  times  faster  is  not  "approximately  the  same  speed".  And 
indeed  the  Big-0  notation  is  not  a  very  precise  way  to  classify  function  convergence  speed,  which  is  why  in 
mathematics  we  use  the  equivalence  relationship  when  we  want  a  precise  estimation  of  speed.  But  for  the 
purposes  of  separating  algorithms  in  large  speed  classes,  Big-0  is  enough.  We  don't  need  to  separate  functions  that 
grow  a  fixed  number  of  times  faster  than  each  other,  but  only  functions  that  grow  infinitely  faster  than  each  other. 
For  instance  if  we  take  h(n)  =  nA2*log(n),  we  see  that  h(n)/g(n)  =  log  (n)  which  tends  to  infinity  with  n  so  h  is 
not  0(nA2),  because  h  grows  infinitely  faster  than  nA2. 

Now  I  need  to  make  a  side  note  :  you  might  have  noticed  that  if  f  =  0(g)  and  g  =  0(h),  then  f  =  0(h).  For 
instance  in  our  case,  we  have  f  =  0(nA3),andf  =  0(nA4)...  In  algorithm  complexity  analysis,  we  frequently  say  f  = 
0(g)  to  mean  that  f  =  0(g)  and  g  =  0(f),  which  can  be  understood  as  "g  is  the  smallest  Big-0  for  f".  In 
mathematics  we  say  that  such  functions  are  Big-Thetas  of  each  other. 

How  is  it  used  ? 

When  comparing  algorithm  performance,  we  are  interested  in  the  number  of  operations  that  an  algorithm 
performs.  This  is  called  time  complexity.  In  this  model,  we  consider  that  each  basic  operation  (addition, 
multiplication,  comparison,  assignment,  etc.)  takes  a  fixed  amount  of  time,  and  we  count  the  number  of  such 
operations.  We  can  usually  express  this  number  as  a  function  of  the  size  of  the  input,  which  we  call  n.  And  sadly, 
this  number  usually  grows  to  infinity  with  n  (if  it  doesn't,  we  say  that  the  algorithm  is  0(1 )).  We  separate  our 
algorithms  in  big  speed  classes  defined  by  Big-0  :  when  we  speak  about  a  "0(nA2)  algorithm",  we  mean  that  the 
number  of  operations  it  performs,  expressed  as  a  function  of  n,  is  a  0(nA2).  This  says  that  our  algorithm  is 
approximately  as  fast  as  an  algorithm  that  would  do  a  number  of  operations  equal  to  the  square  of  the  size  of  its 
input,  or  faster.  The  "or  faster"  part  is  there  because  I  used  Big-0  instead  of  Big-Theta,  but  usually  people  will  say 
Big-0  to  mean  Big-Theta. 
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When  counting  operations,  we  usually  consider  the  worst  case:  for  instance  if  we  have  a  loop  that  can  run  at  most  n 
times  and  that  contains  5  operations,  the  number  of  operations  we  count  is  5n.  It  is  also  possible  to  consider  the 
average  case  complexity. 

Quick  note  :  a  fast  algorithm  is  one  that  performs  few  operations,  so  if  the  number  of  operations  grows  to  infinity 
faster,  then  the  algorithm  is  slower:  O(n)  is  better  than  0(nA2). 

We  are  also  sometimes  interested  in  the  space  complexity  of  our  algorithm.  For  this  we  consider  the  number  of 
bytes  in  memory  occupied  by  the  algorithm  as  a  function  of  the  size  of  the  input,  and  use  Big-0  the  same  way. 

Section  22.1:  A  Simple  Loop 

The  following  function  finds  the  maximal  element  in  an  array: 

int  f ind_max(const  int  *array,  size_t  len)  { 
int  max  =  INT_MIN 

for  (  size_t  i  =  0;  i  <  len  i++)  { 
if  (max  <  array[i  )  { 
max  =  array[i 

} 

} 

return  max; 

} 

The  input  size  is  the  size  of  the  array,  which  I  called  len  in  the  code. 

Let's  count  the  operations. 

int  max  =  INT_MIN 
size_t  i  =  0; 

These  two  assignments  are  done  only  once,  so  that's  2  operations.  The  operations  that  are  looped  are: 

if  (max  <  array[i]) 
i++; 

max  =  array[i] 

Since  there  are  3  operations  in  the  loop,  and  the  loop  is  done  n  times,  we  add  3n  to  our  already  existing  2 
operations  to  get  3n  +  2.  So  our  function  takes  3n  +  2  operations  to  find  the  max  (its  complexity  is  3n  +  2).  This  is 
a  polynomial  where  the  fastest  growing  term  is  a  factor  of  n,  so  it  is  O(n). 

You  probably  have  noticed  that  "operation"  is  notvery  well  defined.  For  instance  I  said  that  if  (max  <  array[i] ) 
was  one  operation,  but  depending  on  the  architecture  this  statement  can  compile  to  for  instance  three  instructions 
:  one  memory  read,  one  comparison  and  one  branch.  I  have  also  considered  all  operations  as  the  same,  even 
though  for  instance  the  memory  operations  will  be  slower  than  the  others,  and  their  performance  will  vary  wildly 
due  for  instance  to  cache  effects.  I  also  have  completely  ignored  the  return  statement,  the  fact  that  a  frame  will  be 
created  for  the  function,  etc.  In  the  end  it  doesn't  matter  to  complexity  analysis,  because  whatever  way  I  choose  to 
count  operations,  it  will  only  change  the  coefficient  of  the  n  factor  and  the  constant,  so  the  result  will  still  be  O(n). 
Complexity  shows  how  the  algorithm  scales  with  the  size  of  the  input,  but  it  isn't  the  only  aspect  of  performance! 

Section  22.2:  A  Nested  Loop 

The  following  function  checks  if  an  array  has  any  duplicates  by  taking  each  element,  then  iterating  over  the  whole 
array  to  see  if  the  element  is  there 
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_Bool  contains_duplicates(const  int  *array,  size_t  len)  { 
for  (int  i  =  0;  i  <  len  -  1  ;  i++)  { 
for  (int  j  =0;  j  <  len  j++)  { 

if  (i  !=  j  &&  array[i  ==  array[j])  { 

return  1 ; 

} 

} 

} 

return  0; 


The  inner  loop  performs  at  each  iteration  a  number  of  operations  that  is  constant  with  n.  The  outer  loop  also  does 
a  few  constant  operations,  and  runs  the  inner  loop  n  times.  The  outer  loop  itself  is  run  n  times.  So  the  operations 
inside  the  inner  loop  are  run  nA2  times,  the  operations  in  the  outer  loop  are  run  n  times,  and  the  assignment  to  i  is 
done  one  time.  Thus,  the  complexity  will  be  something  like  anA2  +  bn  +  c,  and  since  the  highest  term  is  nA2,  the  O 
notation  is  0(nA2). 

As  you  may  have  noticed,  we  can  improve  the  algorithm  by  avoiding  doing  the  same  comparisons  multiple  times. 
We  can  start  from  i  +  1  in  the  inner  loop,  because  all  elements  before  it  will  already  have  been  checked  against  all 
array  elements,  including  the  one  at  index  i  +  i.  This  allows  us  to  drop  the  i  ==  j  check. 


_Bool  f aster_contains_duplicates(const  int  *array,  size_t  len)  { 
for  (int  i  =  0;  i  <  len  1 ;  i++)  { 

for  (int  j  =  i  +  1;  j  <  len  j++)  { 
if  (array [i]  ==  array! j ])  { 

return  1 ; 

} 

} 

} 

return  0; 


Obviously,  this  second  version  does  less  operations  and  so  is  more  efficient.  How  does  that  translate  to  Big-0 
notation?  Well,  now  the  inner  loop  body  is  run  1  +  2  +  ...  +  n  -  1  =  n(n-l  )/2  times.  This  \s  still  a  polynomial  of 
the  second  degree,  and  so  is  still  only  0(nA2).  We  have  clearly  lowered  the  complexity,  since  we  roughly  divided  by 
2  the  number  of  operations  that  we  are  doing,  but  we  are  still  in  the  same  complexity  class  as  defined  by  Big-O.  In 
order  to  lower  the  complexity  to  a  lower  class  we  would  need  to  divide  the  number  of  operations  by  something  that 
tends  to  infinity  with  n. 

Section  22.3:  0(log  n)  types  of  Algorithms 

Let's  say  we  have  a  problem  of  size  n.  Now  for  each  step  of  our  algorithm(which  we  need  write),  our  original 
problem  becomes  half  of  its  previous  size(n/2). 

So  at  each  step,  our  problem  becomes  half. 

Step  Problem 

1  n/2 

2  n/4 

3  n/8 

4  n/1 6 

When  the  problem  space  is  reduced(i.e  solved  completely),  it  cannot  be  reduced  any  further(n  becomes  equal  to  1 ) 
after  exiting  check  condition. 
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1 .  Let's  say  at  kth  step  or  number  of  operations: 


problem-size  =  1 

2.  But  we  know  at  kth  step,  our  problem-size  should  be: 

problem-size  =  n/2k 

3.  From  1  and  2: 

n/2k  =  1  or 
n  =  2k 

4.  Take  logon  both  sides 

loge  n  =  k  loge2 
or 

k  =  loge  n  /  loge  2 

5.  Using  formula  logx  m  /  logx  n  =  logn  m 

k  =  log2  n 
or  simply  k  =  log  n 


Now  we  know  that  our  algorithm  can  run  maximum  up  to  log  n,  hence  time  complexity  comes  as 
0( log  n) 


A  very  simple  example  in  code  to  support  above  text  is  : 

for(int  i=1 ;  i<=n  i=i*2) 

{ 

//  perform  some  operation 

} 

So  now  if  some  one  asks  you  if  n  is  256  how  many  steps  that  loop(  or  any  other  algorithm  that  cuts  down  it's 
problem  size  into  half)  will  run  you  can  very  easily  calculate. 

k  =  log2  256 

k  =  log2  2  8  (  =>  logaa  =  1 ) 
k  =  8 

Another  very  good  example  for  similar  case  is  Binary  Search  Algorithm. 
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int  bSearch(int  arr[],int  size, int  item){ 
int  low=0; 
int  high=size  1  ; 

while ( low<=high) { 

mid=low+(high  low)/2; 
if (arr [mid ]==item) 

return  mid ; 

else  if (arr[mid]<item) 
low=mid+1 ; 
else  high=mid-1 ; 

} 

return  -1 ;//  Unsuccessful  result 

} 

Section  22.4:  An  0(log  n)  example 

Introduction 

Consider  the  following  problem: 

L  is  a  sorted  list  containing  n  signed  integers  (n  being  big  enough),  for  example  [-5,  -2,  -1,  0,  1,  2,  4]  (here,  n 
has  a  value  of  7).  If  L  is  known  to  contain  the  integer  0,  how  can  you  find  the  index  of  0  ? 

Naive  approach 

The  first  thing  that  comes  to  mind  is  to  just  read  every  index  until  0  is  found.  In  the  worst  case,  the  number  of 
operations  is  n,  so  the  complexity  is  O(n). 

This  works  fine  for  small  values  of  n,  but  is  there  a  more  efficient  way  ? 

Dichotomy 

Consider  the  following  algorithm  (Python3): 

a  =  0 
b  =  n-1 

while  True 

h  =  (a+b)//2  ##  //  is  the  integer  division,  so  h  is  an  integer 
if  L[h]  ==  0: 

return  h 

elif  L[h]  >  0: 
b  =  h 

elif  L[h]  <  0: 
a  =  h 

a  and  b  are  the  indexes  between  which  0  is  to  be  found.  Each  time  we  enter  the  loop,  we  use  an  index  between  a 
and  b  and  use  it  to  narrow  the  area  to  be  searched. 

In  the  worst  case,  we  have  to  wait  until  a  and  b  are  equal.  But  how  many  operations  does  that  take?  Not  n,  because 
each  time  we  enter  the  loop,  we  divide  the  distance  between  a  and  b  by  about  two.  Rather,  the  complexity  is  0(log 
n). 

Explanation 

Note:  When  we  write  "log",  we  mean  the  binary  logarithm,  or  log  base  2  (which  we  will  write  "log_2").  As  0(log_2  n)  =  0(1  og 
n)  (you  can  do  the  math)  we  will  use  "log"  instead  of  "log_2". 
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Let's  call  x  the  number  of  operations:  we  know  that  1  =  n  /  (2Ax). 

So  2Ax  =  n,  then  x  =  log  n 

Conclusion 

When  faced  with  successive  divisions  (be  it  by  two  or  by  any  number),  remember  that  the  complexity  is  logarithmic. 
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Chapter  23:  Sorting 


Parameter 


Description 


Average  case  complexity 


Worst  case  complexity 


Stability 


In  place 


Best  case  complexity 


A  sorting  algorithm  is  stable  if  it  preserves  the  relative  order  of  equal  elements  after 
sorting. 

A  sorting  algorithm  is  in-place  if  it  sorts  using  only  0(1 )  auxiliary  memory  (not  counting 
the  array  that  needs  to  be  sorted). 

A  sorting  algorithm  has  a  best  case  time  complexity  of  0(T(n) )  if  its  running  time  is  at 
least  T(n)  for  all  possible  inputs. 

A  sorting  algorithm  has  an  average  case  time  complexity  of  0(T(n) )  if  its  running  time, 

averaged  over  all  possible  inputs,  is  T(n). 

A  sorting  algorithm  has  a  worst  case  time  complexity  of  0(T(n) )  if  its  running  time  is  at 
most  T(n). 


Section  23.1:  Stability  in  Sorting 


Stability  in  sorting  means  whether  a  sort  algorithm  maintains  the  relative  order  of  the  equals  keys  of  the  original 
input  in  the  result  output. 

So  a  sorting  algorithm  is  said  to  be  stable  if  two  objects  with  equal  keys  appear  in  the  same  order  in  sorted  output 
as  they  appear  in  the  input  unsorted  array. 

Consider  a  list  of  pairs: 

(1,  2)  (9,  7)  (3,  4)  (8,  6)  (9,  3) 

Now  we  will  sort  the  list  using  the  first  element  of  each  pair. 

A  stable  sorting  of  this  list  will  output  the  below  list: 

(1,  2)  (3,  4)  (8,  6)  (9,  7)  (9,  3) 

Because  (9,  3)  appears  after  (9,  7)  in  the  original  list  as  well. 

An  unstable  sorting  will  output  the  below  list: 

(1,  2)  (3,  4)  (8,  6)  (9,  3)  (9,  7) 

Unstable  sort  may  generate  the  same  output  as  the  stable  sort  but  not  always. 

Well-known  stable  sorts: 

•  Merge  sort 

•  Insertion  sort 

•  Radix  sort 

•  Tim  sort 

•  Bubble  Sort 

Well-known  unstable  sorts: 

•  Heap  sort 

•  Quick  sort 
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Chapter  24:  Bubble  Sort 

Parameter  Description 

Stable  Yes 

In  place  Yes 

Best  case  complexity  O(n) 

Average  case  complexity  0(nA2) 

Worst  case  complexity  0(nA2) 

Space  complexity  0(1) 

Section  24.1:  Bubble  Sort 

The  BubbleSort  compares  each  successive  pair  of  elements  in  an  unordered  list  and  inverts  the  elements  if  they 
are  not  in  order. 

The  following  example  illustrates  the  bubble  sort  on  the  list  {6, 5, 3, 1 , 8, 7, 2, 4}  (pairs  that  were  compared  in  each 
step  are  encapsulated  in  '**'): 


{6,5, 3, 1,8, 7, 2,4} 
{**5, 6**, 3, 1,8, 7, 2, 4} 
{5, **3,6**, 1,8, 7, 2, 4} 
{5,3,  **1 , 6**, 8, 7, 2, 4} 
{5,3,1,  **6 , 8**, 7, 2, 4} 
{5, 3, 1,6, **7,8**, 2, 4} 
{5, 3, 1 , 6, 7, **2 , 8**, 4} 
{5, 3, 1,6, 7, 2, **4,8**} 


--  5  <  6  ->  swap 
--  3  <  6  ->  swap 
--  1  <  6  ->  swap 
--  8  >  6  ->  no  swap 
--  7  <  8  ->  swap 
--  2  <  8  ->  swap 
--  4  <  8  ->  swap 


After  one  iteration  through  the  list,  we  have  {5, 3, 1,6, 7, 2, 4, 8}.  Note  that  the  greatest  unsorted  value  in  the  array 
(8  in  this  case)  will  always  reach  its  final  position.  Thus,  to  be  sure  the  list  is  sorted  we  must  iterate  n-1  times  for  lists 
of  length  n. 

Graphic: 


6  5  3  1  8  7  2  4 


Section  24.2:  Implementation  in  C  &  C++ 

An  example  implementation  of  BubbleSort  in  C++: 

void  bubbleSort ( vector<int>numbers) 

{ 

for(int  i  =  numbers. size()  -  1;  i  >=  0;  i  -)  { 
for(int  j  =  1 ;  j  <=  i  j++)  { 

if (numbers [ j -1 ]  >  numbers! j])  { 
swap (numbers [j-1 ] , numbers( j ) ) ; 

} 

} 
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} 

} 

C  Implementation 

void  bubble_sort(long  list[],  long  n) 

{ 

long  c,  d,  t; 

for  (c  =  0  ;  c  <  (  n  1  );  C++) 

{ 

for  (d  =  0  ;  d  <  n  c  -  1 ;  d++) 

{ 

if  (list[d]  >  list[d+1]) 

{ 

/*  Swapping  */ 

t  =  list [ d  ]  ; 

list[d]  =  list[d+1 ]  ; 
list [ d+1  ]  =  t  ; 

} 

} 

} 

} 

Bubble  Sort  with  pointer 

void  pointer_bubble_sort(long  *  list,  long  n) 

{ 

long  c,  d,  t; 

for  (c  =  0  ;  c  <  (  n  1  );  C++) 

{ 

for  (d  =  0  ;  d  <  n  c  -  1 ;  d++) 

{ 

if  (  *  (list  +  d  )  >  *(list+d+1)) 

{ 

/*  Swapping  */ 

t  =  *  ( list  +  d  )  ; 

*  (list  +  d  )  =  *  (list  +  d  +  1  ); 

*  ( list  +  d  +  1 )  =  t  ; 

} 

} 

} 

} 

Section  24.3:  Implementation  in  C# 

Bubble  sort  is  also  known  as  Sinking  Sort.  It  is  a  simple  sorting  algorithm  that  repeatedly  steps  through  the  list  to 
be  sorted,  compares  each  pair  of  adjacent  items  and  swaps  them  if  they  are  in  the  wrong  order. 

Bubble  sort  example 
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Implementation  of  Bubble  Sort 

I  used  C#  language  to  implement  bubble  sort  algorithm 

public  class  BubbleSort 

{ 

public  static  void  SortBubble(int[  ]  input) 

{ 

for  (var  i  =  input. Length  -  1;  i  >=  0;  i  -) 

{ 

for  (var  j  =  input. Length  -  1  -  1;  j  >=  0;  j--) 

{ 

if  (input[ j]  <=  input! j  +  1])  continue 
var  temp  =  input [j  +1]; 
input [j  +  1]  =  input[j]; 
input [ j ]  =  temp 

} 

} 

} 

public  static  int[  Main(int[  input) 

{ 

SortBubble( input)  ; 
return  input; 

} 

} 

Section  24.4:  Python  Implementation 

#! /usr/bin/python 

input_list  [10,1  2  11] 

for  i  in  range(len(input_list))  : 
for  j  in  range(i)  : 

if  int( input_list [ j  ] )  >  int(input_list[ j+1 ] ) : 

input_list [ j ]  input_list [ j+1 ]  input_list [ j+1 ]  input_list [ j ] 

print  input_list 
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Section  24.5:  Implementation  in  Java 


public  class  MyBubbleSort  { 

public  static  void  bubble_srt(int  array[])  {//main  logic 
int  n  =  array. length; 

int  k 

for  (int  m  =  n  m  >=  0;  m--)  { 

for  (int  i  =  0;  i  <  n  -  1 ;  i++)  { 
k  =  i  +  1 ; 

if  (array[i  >  array[k])  { 
swapNumbers(i,  k,  array); 

} 

} 

printNumbers(array) ; 

} 

} 

private  static  void  swapNumbers(int  i,  int  j,  int[]  array)  { 

int  temp 
temp  =  array[i] 
array[i)  =  array[ j ] ; 
array[ j ]  =  temp 

} 

private  static  void  printNumbers(int[  input)  { 

for  (int  i  =  0;  i  <  input . length ;  i++)  { 

System. out. print(input[i  +  ",  "); 

} 

System. out . println( " \  n" )  ; 

} 

public  static  void  main (String [ ]  args)  { 

int [ ]  input  =  {  4,  2,  9,  6,  23,  12,  34,  0,  1  }; 
bubble_srt(input) ; 


} 

} 

Section  24.6:  Implementation  in  Javascript 

function  bubbleSort (a) 

{ 

var  swapped; 

do  { 

swapped  =  false 

for  (var  i=0;  i  <  a. length-1;  i++)  { 
if  (a [ i ]  >  a [ i+1 ] )  { 
var  temp  =  a[i 
a [ i ]  =  a [i+1 ]  ; 
a [ i+1 ]  =  temp 
swapped  =  true 

} 

} 

}  while  (swapped) ; 

} 

var  a  =  [3,  203,  34,  746,  200,  984,  198,  764,  9] ; 
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bubbleSort (a) ; 

console. log(a)  ;  //logs  [  3,  9,  34,  7 98,  200,  203,  746,  764,  984  ] 
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Chapter  25:  Merge  Sort 

Section  25.1:  Merge  Sort  Basics 

Merge  Sort  is  a  divide-and-conquer  algorithm.  It  divides  the  input  list  of  length  n  in  half  successively  until  there  are 
n  lists  of  size  1 .  Then,  pairs  of  lists  are  merged  together  with  the  smaller  first  element  among  the  pair  of  lists  being 
added  in  each  step.  Through  successive  merging  and  through  comparison  of  first  elements,  the  sorted  list  is  built. 

An  example: 


Time  Complexity:  T(n)  =  2T(n/2)  +  0(n) 

The  above  recurrence  can  be  solved  either  using  Recurrence  Tree  method  or  Master  method.  It  falls  in  case  II  of 
Master  Method  and  solution  of  the  recurrence  is  0(nLogn).  Time  complexity  of  Merge  Sort  is  0(nLogn)  in  all  3  cases 
(worst,  average  and  best)  as  merge  sort  always  divides  the  array  in  two  halves  and  take  linear  time  to  merge  two 
halves. 

Auxiliary  Space:  0(n) 

Algorithmic  Paradigm:  Divide  and  Conquer 
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Sorting  In  Place:  Not  in  a  typical  implementation 


Stable:  Yes 

Section  25.2:  Merge  Sort  Implementation  in  Go 


package  main 
import  "fmt" 

func  mergeSort(a  [lint)  [ ] int  { 
if  len(a)  <  2  { 

return  a 

} 

m  :=  (len(a))  /  2 

f  :=  mergeSort(a[ :m] ) 
s  :=  mergeSort(a[m: ] ) 

return  merge(f,  s) 


func  merge(f  [lint,  s  []int)  [ ] int  { 
var  i,  j  int 
size  :=  len(f)  +  len(s) 

a  :=  make([[int,  size,  size) 

for  z  :=  0;  z  <  size  z++  { 
lenF  :=  len(f) 
lenS  :=  len(s) 

if  i  >  lenF  1  &&  j  <=  lenS-1  { 
a [ z ]  =  s[j] 

j++ 

}  else  if  j  >  lenS  1  &&  i  <=  lenF-1  { 
a [ z ]  =  f[i] 
i++ 

}  else  if  f [i]  <  s[ j ]  { 
a [ z ]  =  f[i] 
i++ 

}  else  { 

a [ z ]  =  s[j] 

]++ 

} 

} 

return  a 

} 

func  main()  { 

a  :=  [ ] int {75 ,  12,  34,  45,  0,  123,  32,  56,  32,  99,  123,  11,  86,  33} 

fmt . Println(a) 

fmt . Print In (mergeSort (a) ) 


Section  25.3:  Merge  Sort  Implementation  in  C  &  C# 

C  Merge  Sort 
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int  merge(int  arr[],int  l,int  m,int  h) 

{ 

int  arrl [10] , arr2[10]  ;  //  Two  temporary  arrays  to 

hold  the  two  arrays  to  be  merged 

int  nl , n2, i, j , k 

n1=m-l+1  ; 

n2=h-m; 

for(i=0;  i<n1  ;  i++) 
arrl [i]=arr[l+i]  ; 
for(j=0;  j<n2  j++) 
arr2[ j ]=arr[m+j+1 ]  ; 

arrl [ i] =9999 ;  //  To  mark  the  end  of  each  temporary  array 

arr2[ j ] =9999  ; 

i=0  ; 
j=0; 

for(k-l  k<=h  k++)  {  //process  of  combining  two  sorted  arrays 
if (arrl [ i ]<=arr2 [ j  ] ) 
arr[k]=arr1 [i++]  ; 

else 

arr[k]=arr2[ j++] ; 

} 

return  0; 

} 

int  merge_sort(int  arr[  ,  int  low, int  high) 

{ 

int  mid ; 
if(low<high)  { 
mid=( low+high) /2 ; 

//  Divide  and  Conquer 
merge_sort(arr, low, mid)  ; 
merge_sort(arr, mid+1 , high)  ; 

//  Combine 

merge(arr , low, mid ,  high) ; 

} 

return  0; 

} 

C#  Merge  Sort 

public  class  MergeSort 

{ 

static  void  Merge(int[]  input,  int  1,  int  m,  int  r) 

{ 


int 

i,  j; 

var 

nl  = 

m 

-  1  +  1  ; 

var 

n2  = 

r 

-  m; 

var 

left 

= 

new  int[ 

nl]; 

var 

right 

= 

new  int 

[n2]  ; 

for 

(i  = 

0; 

i  <  nl  ; 

i++) 

{ 

left  [ 

i] 

=  input 

[1  +  i 

} 
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for  (j  =  0;  j  <  n2  j++) 

{ 

right[j]  =  input[m  +  j  +  1]; 

} 

i  =  0  ; 
j  =  0; 
var  k  =  1; 

while  (i  <  nl  &&  j  <  n2) 

{ 

if  (left [ i]  <=  right [ j ] ) 

{ 

input [k]  =  left[i] ; 
i++; 

} 

else 

{ 

input[k]  =  right[ j ] ; 

]++; 

} 

k++  ; 

} 

while  ( i  <  nl ) 

{ 

input [k]  =  left[i] ; 

i++; 

k++  ; 

} 

while  (j  <  n2) 

{ 

input[k]  =  right[ j  ]  i 

j++; 

k++; 

} 


static  void  SortMerge(int[ ]  input,  int  1,  int  r) 

{ 

if  (1  <  r) 

{ 

int  m=l+(r-l)/2; 

SortMerge(input,  1,  m) ; 

SortMerge(input,  m  +  1,  r)  ; 

Merge(input,  1,  m,  r)  ; 

} 

} 

public  static  int[]  Main(int[]  input) 

{ 

SortMerge(input,  0,  input. Length  -  1); 
return  input; 

} 


Section  25.4:  Merge  Sort  Implementation  in  Java 

Below  there  is  the  implementation  in  Java  using  a  generics  approach.  It  is  the  same  algorithm,  which  is  presented 
above. 
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public  interface  InPlaceSort<T  extends  Comparable<T>>  { 
void  sort(final  T[]  elements);  } 


public  class  MergeSort  <  T  extends  Comparable  <  T  >>  implements  InPlaceSort  <  T  >  { 


@0verride 

public  void  sort(T[ ]  elements)  { 

T[]  arr  =  ( T [ ] )  new  Comparable[elements. length] ; 
sort(elements,  arr,  0,  elements . length  -  1); 

} 

//  M/e  check  both  our  sides  and  then  merge  them 

private  void  sort(T[]  elements,  T[]  arr,  int  low,  int  high)  { 
if  (low  >=  high)  return 
int  mid  =  low  +  (high  low)  /  2; 
sort(elements,  arr,  low,  mid) ; 
sort(elements,  arr,  mid  +  1,  high); 
merge(elements,  arr,  low,  high,  mid); 

} 


private  void  merge(T[ ]  a,  T[]  b,  int  low,  int  high,  int  mid)  { 
int  i  =  low 
int  j  =  mid  +  1  ; 


//  We  select  the  smallest  element  of  the  two.  And  then  we  put  it  into  b 
for  (int  k  =  low  k  <=  high  k++)  { 

if  (i  <=  mid  &&  j  <=  high)  { 

if  (a[i] . compareTo(a[ j  ] )  >=  0)  { 
b[k]  =  a[ j++]  ; 

}  else  { 

b[k]  =  a[i++]  ; 

} 

}  else  if  (j  >  high  &&  i  <=  mid)  { 
b [ k ]  =  a [ i++ ]  ; 

}  else  if  (i  >  mid  &&  j  <=  high)  { 
b[k]  =  a[ j++]  ; 

} 

} 

for  (int  n  =  low  n  <=  high  n++)  { 
a [ n ]  =  b[n] ; 

>}} 

Section  25.5:  Merge  Sort  Implementation  in  Python 


def  merge(X,  Y) : 

"  merge  two  sorted  lists  " 
pi  =  p2  =  0 
out  =  [ ] 

while  pi  <  len(X)  and  p2  <  len(Y) : 
if  X [ pi  ]  <  Y [ p2  ]  : 

out .append(X[p1 ] ) 
pi  +=  1 

else 

out . append( Y[ p2 ] ) 

p2  +=  1 

out  +=  X [ pi : ]  +  Y[p2 : ] 
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return  out 


def  mergeSort(A)  : 
if  len(A)  <=  1  : 

return  A 
if  len(A)  ==  2: 

return  sorted(A) 

mid  =  len(A)  /  2 

return  merge(mergeSort(A[ :mid] ) ,  mergeSort(A[mid  :  ] ) ) 

if _ name _ ==  "__main__"  : 

#  Generate  20  random  numbers  and  sort  them 
A  =  [randint(1,  100)  for  i  in  xrange(20)] 
print  mergeSort(A) 

Section  25.6:  Bottoms-up  Java  Implementation 

public  class  MergeSortBU  { 

private  static  Integer[]  array  =  {  4,  3,  1,  8,  9,  15,  20,  2,  5,  6,  30,  70, 

60,80,0,9,67,54,51,52,24,54,7  }; 

public  MergeSortBU( )  { 

} 

private  static  void  merge(Comparable[  ]  arrayToSort,  Comparable)]  aux,  int  lo,int  mid,  int  hi)  { 

for  (int  index  =  0;  index  <  arrayToSort . length ;  index++)  { 
aux[index]  =  arrayToSort[index]  ; 

} 


int  i  =  lo 
int  j  =  mid  +  1  ; 

for  (int  k  =  lo  k  <=  hi  k++)  { 
if  (i  >  mid) 

arrayToSort [k  =  aux[j++]; 
else  if  (j  >  hi) 

arrayToSort [k  =  aux[i++]; 
else  if  (isLess(aux[i  ,  aux[j]))  { 
arrayToSort [k  =  aux[i++]; 

}  else  { 

arrayToSort [k  =  aux[j++]; 

} 

} 

} 

public  static  void  sort(Comparable[  ]  arrayToSort,  Comparable)]  aux,  int  lo,  int  hi)  { 
int  N  =  arrayToSort . length ; 
for  (int  sz  =  1 ;  sz  <  N  sz  =  sz  +  sz)  { 

for  (int  low  =  0;  low  <  N  low  =  low  +  sz  +  sz)  { 

System. out . println( "Size : "+  sz) ; 

merge(arrayToSort,  aux,  low,  low  +  sz  -1  , Math .min( low  +  sz  +  sz  -  1 ,  N  -  1 ) )  ; 
print (arrayToSort) ; 

} 

} 


} 

public  static  boolean  isLess(Comparable  a,  Comparable  b)  { 
return  a . compareTo(b)  <=  0; 
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} 

private  static  void  print(Comparable[ ]  array) 

{http :/ /stackoverf low .com /documentation /algorithm/ 5732 /merge -sort# 

StringBuffer  buffer  =  new 

StringBuf fer( )  http  //stackoverf low. com/documentation/algorithm/ 5732/merge-sort# 
for  (Comparable  value  array)  { 
buffer.append(value); 
buffer . append ( '  '  )  ; 

} 

System. out. println(buffer) ; 


public  static  void  main (String [ ]  args)  { 

Comparable!]  aux  =  new  Comparable[array . length] ; 
print(array) ; 

MergeSortBU.sort(array,  aux,  0,  array. length  -  1); 

} 

} 
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Chapter  26:  Insertion  Sort 

Section  26.1:  Haskell  Implementation 

insertSort  : :  Ord  a  =>  [a  ->  [a 
insertSort  [ ]  =  [  ] 

insertSort  (x:xs)  =  insert  x  (insertSort  xs) 

insert  :  Ord  a  =>  a->  [a  ->  [a 
insert  n  [ ]  =  [ n 

insert  n  (x:xs)  |  n  <=  x  =  (n  x:xs) 

|  otherwise  =  x:insert  n  xs 
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Chapter  27:  Bucket  Sort 

Section  27.1:  C#  Implementation 

public  class  BucketSort 

{ 

public  static  void  SortBucket ( ref  int[  input) 

{ 

int  minValue  =  input[0]; 
int  maxValue  =  input[0]; 
int  k  =  0  ; 

for  (int  i  =  input. Length  -  1;  i  >=  1  i  -) 

{ 

if  (input[i  >  maxValue)  maxValue  =  input[i 
if  (input[i|  <  minValue)  minValue  =  input[i 

} 

List<int>[]  bucket  =  new  List<int>[maxValue  -  minValue  +  1]; 

for  (int  i  =  bucket . Length  -  1;  i  >=  0;  i--) 

{ 

bucket[i  =  new  List<int>(); 

} 

foreach  (int  i  in  input) 

{ 

bucket[i  -  minValue  .Add(i); 

} 

foreach  (List<int>  b  in  bucket) 

{ 

if  (b. Count  >  0) 

{ 

foreach  (int  t  in  b) 

{ 

input [k  =  t; 

k++; 

} 

} 

} 

} 

public  static  int[  Main(int[|  input) 

{ 

SortBucket( ref  input); 
return  input; 

} 

} 
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Chapter  28:  Quicksort 

Section  28.1:  Quicksort  Basics 

Quicksort  is  a  sorting  algorithm  that  picks  an  element  ("the  pivot")  and  reorders  the  array  forming  two  partitions 
such  that  all  elements  less  than  the  pivot  come  before  it  and  all  elements  greater  come  after.  The  algorithm  is  then 
applied  recursively  to  the  partitions  until  the  list  is  sorted. 

1.  Lomuto  partition  scheme  mechanism  : 

This  scheme  chooses  a  pivot  which  is  typically  the  last  element  in  the  array.  The  algorithm  maintains  the  index  to 
put  the  pivot  in  variable  i  and  each  time  it  finds  an  element  less  than  or  equal  to  pivot,  this  index  is  incremented 
and  that  element  would  be  placed  before  the  pivot. 

partition (A,  low,  high)  is 
pivot  :=  A[high 
i  :=  low 

for  j  :=  low  to  high  -  1  do 
if  A[ j ]  <  pivot  then 
swap  A[ i  with  A[ j ] 

i  :=  i  +  1 

swap  A [ i  with  A[high 

return  i 

Quick  Sort  mechanism  : 

quicksort(A,  low,  high)  is 
if  low  <  high  then 

p  :=  partition(A,  low,  high) 
quicksort(A,  low,  p  -  1) 
quicksort(A,  p  +  1,  high) 

Example  of  quick  sort: 
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2.  Hoare  partition  scheme: 

It  uses  two  indices  that  start  at  the  ends  of  the  array  being  partitioned,  then  move  toward  each  other,  until  they 
detect  an  inversion:  a  pair  of  elements,  one  greater  or  equal  than  the  pivot,  one  lesser  or  equal,  that  are  in  the 
wrong  order  relative  to  each  other.  The  inverted  elements  are  then  swapped.  When  the  indices  meet,  the  algorithm 
stops  and  returns  the  final  index.  Hoare's  scheme  is  more  efficient  than  Lomuto's  partition  scheme  because  it  does 
three  times  fewer  swaps  on  average,  and  it  creates  efficient  partitions  even  when  all  values  are  equal. 


quicksort(A,  lo,  hi)  is 
if  lo  <  hi  then 

p  :=  partition(A,  lo,  hi) 
quicksort(A,  lo,  p) 
quicksort(A,  p  +  1,  hi) 

Partition  : 


partition(A,  lo,  hi)  is 
pivot  :=  A[lo 

i  :=  lo  -  1 
j  :=  hi  +  1 

loop  forever 

do 

i  :=  i  +  1 


GoalKicker.com  -  Algorithms  Notes  for  Professionals 


140 


while  A [ i  <  pivot  do 

do 

j  :=  j  -  1 

while  A[j]  >  pivot  do 

if  i  >=  j  then 

return  j 

swap  A [ i ]  with  A[ j ] 

Section  28.2:  Quicksort  in  Python 

def  quicksort(arr) : 
if  len(arr)  <=  1  : 

return  arr 

pivot  =  arr[len(arr)  /  2] 

left  =  [x  for  x  in  arr  if  x  <  pivot] 

middle  =  [x  for  x  in  arr  if  x  ==  pivot] 

right  =  [x  for  x  in  arr  if  x  >  pivot] 

return  quicksort(left)  +  middle  +  quicksort( right) 

print  quicksort ([3,6,8, 10,1 ,2,1 ]) 

Prints  "[1,1,2,  3,  6,  8,10]" 

Section  28.3:  Lomuto  partition  java  implementation 

public  class  Solution  { 

public  static  void  main(String[  ]  args)  { 

Scanner  sc  =  new  Scanner(System . in)  ; 
int  n  =  sc . nextlnt () ; 
int[ ]  ar  =  new  int[n]  ; 
for (int  i=0;  i<n  i++) 
ar  [  i  =  sc . nextlnt ( )  ; 
quickSort(ar,  0,  ar. length-1); 


public  static  void  quickSort(int[ ]  ar,  int  low,  int  high) 

{ 

if ( low<high) 

{ 

int  p  =  partition(ar,  low,  high); 
quickSort(ar,  0  ,  p-1); 
quickSort(ar,  p+1 ,  high); 

} 

} 

public  static  int  partition(int[ ]  ar,  int  1,  int  r) 

{ 

int  pivot  =  ar[  r] ; 

int  i  =1 

for(int  j=l  j<r;  ]++) 

{ 

if (ar [j ]  <=  pivot) 

{ 

int  t  =  ar[ j ] ; 
ar [ j ]  =  ar[ i] ; 
ar [ i]  =  t; 
i++  ; 

} 
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} 

int  t  =  ar[i] ; 
a  r  [  i  ]  =  a  r  [  r  ]  ; 
ar[r]  =  t; 

return  i 

} 
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Chapter  29:  Counting  Sort 

Section  29.1:  Counting  Sort  Basic  Information 

Counting  sort  is  an  integer  sorting  algorithm  for  a  collection  of  objects  that  sorts  according  to  the  keys  of  the 
objects. 


Steps 

1 .  Construct  a  working  array  C  that  has  size  equal  to  the  range  of  the  input  array  A. 

2.  Iterate  through  A,  assigning  C[x]  based  on  the  number  of  times  x  appeared  in  A. 

3.  Transform  C  into  an  array  where  C[x]  refers  to  the  number  of  values  <  x  by  iterating  through  the  array, 
assigning  to  each  C[x]  the  sum  of  its  prior  value  and  all  values  in  C  that  come  before  it. 

4.  Iterate  backwards  through  A,  placing  each  value  in  to  a  new  sorted  array  B  at  the  index  recorded  in  C.  This  is 
done  for  a  given  A[x]  by  assigning  B[C[A[x]]]  toA[x],  and  decrementing  C[A[x]]  in  case  there  were  duplicate 
values  in  the  original  unsorted  array. 


Example  of  Counting  Sort 
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Auxiliary  Space:  0(n+k) 

Time  Complexity:  Worst-case:  0(n+k),  Best-case:  0(n),  Average-case  0(n+k) 


Section  29.2:  Psuedocode  Implementation 

Constraints: 


1.  Input  (an  array  to  be  sorted) 

2.  Number  of  element  in  input  (n) 

3.  Keys  in  the  range  of  0..k-1  (k) 

4.  Count  (an  array  of  number) 

Pseudocode: 


for  x  in  input 

count [key(x) ]  +=  1 
total  =  0 

for  i  in  range(k) : 

oldCount  =  count[i] 
countji]  =  total 
total  +=  oldCount 
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for  x  in  input 

output [ count [ key(x) ] ]  =  x 
count [key(x) ]  +=  1 
return  output 
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Chapter  30:  Heap  Sort 

Section  30.1:  C#  Implementation 

public  class  HeapSort 

{ 

public  static  void  Heapify(int[ ]  input,  int  n,  int  i) 

{ 

int  largest  =  i 
int  1  =  i  +  1  ; 
int  r  =  i  +  2; 

if  (1  <  n  &&  input[l  >  input [ largest ] ) 
largest  =  1 

if  (r  <  n  &&  input[r]  >  input [ largest  ] ) 
largest  =  r; 

if  (largest  !=  i) 

{ 

var  temp  =  input[i 
input[i]  =  input[largest]  ; 
input [ largest ]  =  temp 
Heapify(input,  n,  largest); 

} 

} 

public  static  void  SortHeap(int[  input,  int  n) 

{ 

for  (var  i  =  n  1;  i  >=  0;  i — ) 

{ 

Heapify(input,  n,  i) ; 

} 

for  (int  j  =  n  -  1;  j  >=  0;  j--) 

{ 

var  temp  =  input [ 0]  ; 
input[0]  =  input[ j  ]  ; 
input [ j ]  =  temp 
Heapify(input,  j,  0); 

} 

} 

public  static  int[  Main(int[  input) 

{ 

SortHeap(input,  input .Length) ; 
return  input; 

} 

} 

Section  30.2:  Heap  Sort  Basic  Information 

Heap  sort  is  a  comparison  based  sorting  technique  on  binary  heap  data  structure.  It  is  similar  to  selection  sort  in 
which  we  first  find  the  maximum  element  and  put  it  at  the  end  of  the  data  structure.  Then  repeat  the  same  process 
for  the  remaining  items. 

Pseudo  code  for  Heap  Sort: 

function  heapsort(input,  count) 
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heapify(a, count) 
end  <-  count  -  1 
while  end  ->  0  do 
swap(a[end] , a[0] ) 
end<-end-1 
restore(a,  0,  end) 

function  heapify(a,  count) 

start  <-  parent(count  -  1) 
while  start  >=  0  do 

restore(a,  start,  count  -  1) 
start  <-  start  -  1 

Example  of  Heap  Sort: 

Examples  The  fig.  shows  steps  of  heap-sort  for  list  (23  7  1  8  5  6) 


Auxiliary  Space:  0(1) 

Time  Complexity:  O(nlogn) 


GoalKicker.com  -  Algorithms  Notes  for  Professionals 


146 


Chapter  31:  Cycle  Sort 

Section  31.1:  Pseudocode  Implementation 


( input) 
output  =  0 

for  cycleStart  from  0  to  length (array)  -  2 
item  =  array [ cycleStart ] 
pos  =  cycleStart 

for  i  f  rom  cycleStart  +  1  to  length(array)  -  1 
if  array[i]  <  item: 
pos  +=  1 

if  pos  ==  cycleStart: 

continue 

while  item  ==  array[pos] : 
pos  +=  1 

array[pos],  item  =  item,  array[pos 
writes  +=  1 

while  pos  !=  cycleStart: 
pos  =  cycleStart 

for  i  f  rom  cycleStart  +  1  to  length(array)  -  1 
if  array[i]  <  item: 
pos  +=  1 

while  item  ==  array[pos] 
pos  +=  1 

array[pos],  item  =  item,  array[pos] 
writes  +=  1 
return  outout 
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Chapter  32:  Odd-Even  Sort 

Section  32.1:  Odd-Even  Sort  Basic  Information 

An  Odd-Even  Sort  or  brick  sort  is  a  simple  sorting  algorithm,  which  is  developed  for  use  on  parallel  processors  with 
local  interconnection.  It  works  by  comparing  all  odd/even  indexed  pairs  of  adjacent  elements  in  the  list  and,  if  a  pair 
is  in  the  wrong  order  the  elements  are  switched.  The  next  step  repeats  this  for  even/odd  indexed  pairs.  Then  it 
alternates  between  odd/even  and  even/odd  steps  until  the  list  is  sorted. 

Pseudo  code  for  Odd-Even  Sort: 


if  n>2  then 

1.  apply  odd-even  merge(n/2)  recursively  to  the  even  subsequence  a0,  a2,  ...,  an-2  and  to  the 
odd  subsequence  al ,  a3,  ,  ...,  an  1 

2.  comparison  [i  i+1 ]  for  all  i  element  {1,  3,  5,  7,  ...,  n-3} 

else 

comparison  [0:1] 


Wikipedia  has  best  illustration  of  Odd-Even  sort: 


Example  of  Odd-Even  Sort: 
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Implementation: 

I  used  C#  language  to  implement  Odd-Even  Sort  Algorithm. 

public  class  OddEvenSort 

{ 

private  static  void  SortOddEven(int[ ]  input,  int  n) 

{ 

var  sort  =  false 

while  ( ! sort) 

{ 

sort  =  true 

for  (var  i  =  1 ;  i  <  n  -  1 ;  i  +=  2) 

{ 

if  (input[i|  <=  input[i  +  1])  continue 

var  temp  =  input[i 

input [i  =  input [i  +  1 ]  ; 

input [ i  +  1 ]  =  temp 

sort  =  false 

} 

for  (var  i  =  0;  i  <  n  -  1 ;  i  +=  2) 

{ 

if  (input[ii  <=  input[i  +  1])  continue 

var  temp  =  input[i 

input [i  =  input [i  +1]; 

input [ i  +  1 ]  =  temp 

sort  =  false 

} 
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} 


} 


public  static  int[  Main(int[  input) 

{ 

SortOddEven(input,  input. Length)  ; 
return  input; 

} 

} 

Auxiliary  Space:  0(n) 

Time  Complexity:  0(n) 
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Chapter  33:  Selection  Sort 

Section  33.1:  Elixir  Implementation 


defmodule  Selection  do 

def  sort(list)  when  is_list(list)  do 
do_selection(list,  []) 
end 

def  do_selection( [head | [ ]  ,  acc)  do 
acc  ++  [head] 
end 

def  do_selection( list ,  acc)  do 
min  =  min(list) 

do_selection( : lists .delete(min,  list),  acc  ++  [min|) 
end 

defp  min( [first | [second  |  []]  ])  do 
smaller(first,  second) 
end 

defp  min( [first | [second | tail  ])  do 
min( [smaller (first,  second) | t ail ] ) 
end 

defp  smaller(e1,  e2)  do 
if  el  <=  e2  do 
el 
else 
e2 
end 
end 
end 

Selection . sort([100,4,10,6,9,3]) 

|>  10. inspect 

Section  33.2:  Selection  Sort  Basic  Information 

Selection  sort  is  a  sorting  algorithm,  specifically  an  in-place  comparison  sort.  It  has  0(n2)  time  complexity,  making  it 
inefficient  on  large  lists,  and  generally  performs  worse  than  the  similar  insertion  sort.  Selection  sort  is  noted  for  its 
simplicity,  and  it  has  performance  advantages  over  more  complicated  algorithms  in  certain  situations,  particularly 
where  auxiliary  memory  is  limited. 

The  algorithm  divides  the  input  list  into  two  parts:  the  sublist  of  items  already  sorted,  which  is  built  up  from  left  to 
right  at  the  front  (left)  of  the  list,  and  the  sublist  of  items  remaining  to  be  sorted  that  occupy  the  rest  of  the  list. 
Initially,  the  sorted  sublist  is  empty  and  the  unsorted  sublist  is  the  entire  input  list.  The  algorithm  proceeds  by 
finding  the  smallest  (or  largest,  depending  on  sorting  order)  element  in  the  unsorted  sublist,  exchanging  (swapping) 
it  with  the  leftmost  unsorted  element  (putting  it  in  sorted  order),  and  moving  the  sublist  boundaries  one  element  to 
the  right. 

Pseudo  code  for  Selection  sort: 

function  select(list[1 . .n] ,  k) 
for  i  from  1  to  k 
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minlndex  =  i 
minValue  =  list[i] 
for  j  from  i+1  to  n 

if  list [ j ]  <  minValue 
minlndex  =  j 
minValue  =  list[j] 
swap  list[i  and  list[minlndex] 
return  list[k 

Visualization  of  selection  sort: 


Example  of  Selection  sort: 
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Auxiliary  Space:  0(n) 

Time  Complexity:  0(nA2) 

Section  33.3:  Implementation  of  Selection  sort  in  C# 

I  used  C#  language  to  implement  Selection  sort  algorithm. 

public  class  SelectionSort 

{ 

private  static  void  SortSelection(int[  input,  int  n) 

{ 

for  (int  i  =  0;  i  <  n  -  1 ;  i++) 

{ 

var  minld  =  i 

int  j ; 

for  (j  =  i  +  1;  j  <  n  j++) 

{ 

if  (input[j]  <  input[minld] )  minld  =  j; 

} 

var  temp  =  input[minld]  ; 
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input [minld ]  =  input[i 
input [i]  =  temp 

} 

} 

public  static  int[]  Main(int[]  input) 

{ 

SortSelection(input,  input . Length)  ; 
return  input; 

} 
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Chapter  34:  Trees 

Section  34.1:  Typical  anary  tree  representation 

Typically  we  represent  an  anary  tree  (one  with  potentially  unlimited  children  per  node)  as  a  binary  tree,  (one  with 
exactly  two  children  per  node).  The  "next"  child  is  regarded  as  a  sibling.  Note  that  if  a  tree  is  binary,  this 
representation  creates  extra  nodes. 

We  then  iterate  over  the  siblings  and  recurse  down  the  children.  As  most  trees  are  relatively  shallow  -  lots  of 
children  but  only  a  few  levels  of  hierarchy,  this  gives  rise  to  efficient  code.  Note  human  genealogies  are  an 
exception  (lots  of  levels  of  ancestors,  only  a  few  children  per  level). 

If  necessary  back  pointers  can  be  kept  to  allow  the  tree  to  be  ascended.  These  are  more  difficult  to  maintain. 

Note  that  it  is  typical  to  have  one  function  to  call  on  the  root  and  a  recursive  function  with  extra  parameters,  in  this 
case  tree  depth. 

struct  node 

{ 

struct  node  *next; 
struct  node  *child ; 
std :  :string  data 

} 

void  printtree_r(struct  node  *node,  int  depth) 

{ 

int  i 

while(node) 

{ 

if (node->child) 

{ 

for(i=0  i<depth*3  i++) 
printf ( "  "); 
printf ( " {\n" )  : 

printtree_r (node  >child,  depth  +1 )  ; 
for(i=0  i<depth*3  i++) 
printf ("  "); 
printf ( " {\n" )  : 

for(i=0  i<depth*3  i++) 
printf ("  "); 

printf ( "%s\n" ,  node->data . c_str( ) )  ; 
node  =  node  >next ; 

} 

} 

} 

void  printtree(node  *root) 

{ 

printree_r( root,  0); 

} 

Section  34.2:  Introduction 

Trees  are  a  sub-type  of  the  more  general  node-edge  graph  data  structure. 
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To  be  a  tree,  a  graph  must  satisfy  two  requirements: 


•  It  is  acyclic.  It  contains  no  cycles  (or  "loops"). 

•  It  is  connected.  For  any  given  node  in  the  graph,  every  node  is  reachable.  All  nodes  are  reachable  through 
one  path  in  the  graph. 

The  tree  data  structure  is  quite  common  within  computer  science.  Trees  are  used  to  model  many  different 
algorithmic  data  structures,  such  as  ordinary  binary  trees,  red-black  trees,  B-trees,  AB-trees,  23-trees,  Heap,  and 
tries. 

it  is  common  to  refer  to  a  Tree  as  a  Rooted  T ree  by: 

choosing  1  cell  to  be  called  'Root' 
painting  the  'Root'  at  the  top 

creating  lower  layer  for  each  cell  in  the  graph  depending  on  their  distance  from  the  root  -the 
bigger  the  distance,  the  lower  the  cells  (example  above) 

common  symbol  for  trees:  T 

Section  34.3:  To  check  if  two  Binary  trees  are  same  or  not 

1.  For  example  if  the  inputs  are: 

Examples 

a) 
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Output  should  be  true. 
Example:2 

If  the  inputs  are: 
a) 


b) 


Output  should  be  false. 

Pseudo  code  for  the  same: 

boolean  sameTree(node  rootl ,  node  root2){ 
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if ( rootl  ==  NULL  &&  root2  ==  NULL) 
return  true 

if (rootl  ==  NULL  ||  root2  ==  NULL) 
return  false 

if ( rootl ->data  ==  root2  >data 

&&  sameTree( rootl ->left, root2->left) 

&&  sameTree(root1->right,  root2  >right)) 

return  true 

} 
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Chapter  35:  Binary  Search  Trees 

Binary  tree  is  a  tree  that  each  node  in  it  has  maximum  of  two  children.  Binary  search  tree  (BST)  is  a  binary  tree 
which  its  elements  positioned  in  special  order.  In  each  BST  all  values(i.e  key)  in  left  sub  tree  are  less  than  values  in 
right  sub  tree. 

Section  35.1:  Binary  Search  Tree  -  Insertion  (Python) 

This  is  a  simple  implementation  of  Binary  Search  Tree  Insertion  using  Python. 

An  example  is  shown  below: 


www.penjee.com 

Following  the  code  snippet  each  image  shows  the  execution  visualization  which  makes  it  easier  to  visualize  how  this 
code  works. 

class  Node 

def  __init__(self ,  val) : 
self.l_child  =  None 
self.r_child  =  None 
self. data  =  val 


Frames 


Objects 


Global  frame 

Node  class 
hide  attributes 

Node  • — 

_ init _ 

function 

init  (self,  val) 

def  insert( root,  node): 
if  root  is  None 
root  =  node 

else 

if  root. data  >  node. data: 
if  root.l_child  is  None 
root.l_child  =  node 

else 

insert ( root . l_child ,  node) 

else 

if  root.r_child  is  None 
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root.r_child  =  node 

else 

insert( root . r_child ,  node) 


Frames 


Objects 


Global  frame 
Node 
insert 


Node  class 
hide  attributes 


def  in_order_print( root) : 
if  not  root : 

return 

in_order_print( root . l_child) 
print  root. data 
in_order_print( root . r_child) 


Frames  Objects 


Global  frame 
Node 
insert 
in_order_print 


Node  class 
hide  attributes 


def  pre_order_print ( root)  : 
if  not  root : 

return 

print  root. data 
pre_order_print(root.l_ child) 
pre_order_print( root . r_child) 
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Frames 


Objects 


Global  frame 
Node 
insert 
in_order_print 
preorderprint 


Node  class 
hide  attributes 


init 


function 

init  (self,  val) 


function 

insert (root,  node) 

function 

in  order  print ( root) 

function 

pre  order  print( root) 


Section  35.2:  Binary  Search  Tree  -  Deletion(C++) 

Before  starting  with  deletion  I  just  want  to  put  some  lights  on  what  is  a  Binary  search  tree(BST),  Each  node  in  a  BST 
can  have  maximum  of  two  nodes(left  and  right  child).The  left  sub-tree  of  a  node  has  a  key  less  than  or  equal  to  its 
parent  node's  key.  The  right  sub-tree  of  a  node  has  a  key  greater  than  to  its  parent  node's  key. 

Deleting  a  node  in  a  tree  while  maintaining  its  Binary  search  tree  property. 


There  are  three  cases  to  be  considered  while  deleting  a  node. 

•  Case  1 :  Node  to  be  deleted  is  the  leaf  node. (Node  with  value  22). 

•  Case  2:  Node  to  be  deleted  has  one  child. (Node  with  value  26). 

•  Case  3:  Node  to  be  deleted  has  both  children. (Node  with  value  49). 

Explanation  of  cases: 

1 .  When  the  node  to  be  deleted  is  a  leaf  node  then  simply  delete  the  node  and  pass  nullptr  to  its  parent  node. 

2.  When  a  node  to  be  deleted  is  having  only  one  child  then  copy  the  child  value  to  the  node  value  and  delete 
the  child  (Converted  to  case  1) 

3.  When  a  node  to  be  delete  is  having  two  childs  then  the  minimum  from  its  right  sub  tree  can  be  copied  to  the 
node  and  then  the  minimum  value  can  be  deleted  from  the  node's  right  subtree  (Converted  to  Case  2) 

Note:  The  minimum  in  the  right  sub  tree  can  have  a  maximum  of  one  child  and  that  too  right  child  if  it's  having  the 
left  child  that  means  it's  not  the  minimum  value  or  it's  not  following  BST  property. 
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The  structure  of  a  node  in  a  tree  and  the  code  for  Deletion: 


struct  node 

{ 

int  data 

node  *left,  *right; 

}; 


node*  delete_node(node  *root,  int  data) 

{ 

if (root  ==  nullptr)  return  root 

else  if (data  <  root->data)  root->left  =  delete_node( root->left ,  data); 
else  if (data  >  root->data)  root->right  =  delete_node( root->right,  data); 

else 

{ 

if ( root->left  ==  nullptr  &&  root->right  ==  nullptr)  //  Case  7 
{ 

f ree( root) ; 
root  =  nullptr; 

> 

else  if ( root->lef t  ==  nullptr)  //  Case  2 

{ 

node*  temp  =  root; 
root=  root->right; 
f ree(temp) ; 

} 

else  if ( root->right  ==  nullptr)  //  Case  2 

{ 

node*  temp  =  root; 
root  =  root->left; 
f ree(temp) ; 

} 

else  //  Case  3 

{ 

node*  temp  =  root->right; 

while(temp ->left  !=  nullptr)  temp  =  temp->left 
root->data  =  temp  >data 

root->right  =  delete_node( root->right,  temp->data) ; 

} 

} 

return  root 

} 

Time  complexity  of  above  code  is  O (h),  where  h  is  the  height  of  the  tree. 

Section  35.3:  Lowest  common  ancestor  in  a  BST 

Consider  the  BST: 
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Lowest  common  ancestor  of  22  and  26  is  24 
Lowest  common  ancestor  of  26  and  49  is  46 
Lowest  common  ancestor  of  22  and  24  is  24 

Binary  search  tree  property  can  be  used  for  finding  nodes  lowest  ancestor 

Psuedo  code: 

lowestCommonAncestor( root , nodel ,  node2) { 

if (root  ==  NULL) 
return  NULL; 


else  if (nodel ->data  ==  root->data  ||  node2 ->data==  root->data) 
return  root ; 

else  if ( (nodel ->data  <=  root->data  &&  node2  >data  >  root->data) 

||  (node2  ->data  <=  root->data  &&  node1->data  >  root->data) ) { 

return  root; 

} 

else  if ( root->data  >  max(node1 ->data, node2 ->data) ) { 

return  lowestCommonAncestor(root->left,  nodel,  node2)  ; 

} 


else  { 

return  lowestCommonAncestor ( root->right ,  nodel,  node2) ; 

} 

} 

Section  35.4:  Binary  Search  Tree  -  Python 


class  Node(object) : 

def  __init__(self ,  val)  : 
self.l_child  =  None 
self.r_child  =  None 
self. val  =  val 


class  BinarySearchTree(object) : 
def  insert(self,  root,  node): 
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if  root  is  None 
return  node 

if  root.val  <  node.val: 

root.r_child  =  self . insert( root . r_child ,  node) 

else 

root.l_child  =  self . insert ( root . l_child ,  node) 
return  root 


def  in_order_place(self ,  root): 
if  not  root : 

return  None 

else 

self . in_order_place( root . 1_ child) 
print  root.val 

self . in_order_place( root . r_child) 

def  pre_order_place(self ,  root) : 
if  not  root : 

return  None 

else 

print  root.val 

self . pre_order _ place) root . l_child) 
self . pre_order_ place) root . r_child) 

def  post_order_place(self ,  root): 
if  not  root : 

return  None 

else 

self . post _order_place( root . l_child) 
self . post_order_place( root . r_child) 
print  root.val 

"""  Create  different  node  and  insert  data  into  it""" 

r  =  Node(3) 

node  =  BinarySearchTree)) 

nodeList  =  [1,  8,  5,  12,  14,  6,  15,  7,  16,  8] 


for  nd  in  nodeList: 

node . insert ( r,  Node(nd)) 


print 

11 

--In  order  - 

print 

(node . 

. in_order_place(r)) 

print 

" 

--Pre  order  - 

print 

(node . 

. pre_order_place(r)) 

print 

" 

--Post  order  - 

print 

(node . 

. post_order_place(r) 
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Chapter  36:  Check  if  a  tree  is  BST  or  not 

Section  36.1:  Algorithm  to  check  if  a  given  binary  tree  is  BST 

A  binary  tree  is  BST  if  it  satisfies  any  one  of  the  following  condition: 

1.  It  is  empty 

2.  It  has  no  subtrees 

3.  For  every  node  x  in  the  tree  all  the  keys  (if  any)  in  the  left  sub  tree  must  be  less  than  key(x)  and  all  the  keys  (if 
any)  in  the  right  sub  tree  must  be  greater  than  key(x). 

So  a  straightforward  recursive  algorithm  would  be: 

is_BST( root) : 
if  root  ==  NULL: 
return  true 

//  Check  values  in  left  subtree 

if  root->left  !=  NULL: 

max_key_in_left  =  f ind_max_key( root->left) 
if  max_key_in_lef t  >  root->key 

return  false 

//  Check  values  in  right  subtree 
if  root->right  !=  NULL 

min_key_in_right  =  find_min_key( root->right) 
if  min_key_in_right  <  root->key 

return  false 

return  is_BST( root->lef t)  &&  is_BST( root->right) 

The  above  recursive  algorithm  is  correct  but  inefficient,  because  it  traverses  each  node  mutiple  times. 

Another  approach  to  minimize  the  multiple  visits  of  each  node  is  to  remember  the  min  and  max  possible  values  of 
the  keys  in  the  subtree  we  are  visiting.  Let  the  minimum  possible  value  of  any  key  be  K_MIN  and  maximum  value  be 
K_MAX.  When  we  start  from  the  root  of  the  tree,  the  range  of  values  in  the  tree  is  [K_MIN,  K_MAX].  Let  the  key  of  root 
node  be  x.  Then  the  range  of  values  in  left  subtree  is  [  K_MIN,  x)  and  the  range  of  values  in  right  subtree  is 
(x,  K_MAX].  We  will  use  this  idea  to  develop  a  more  efficient  algorithm. 

is_BST( root ,  min,  max): 
if  root  ==  NULL: 
return  true 

//  is  the  current  node  key  out  of  range? 

if  root->key  <  min  ||  root->key  >  max: 

return  false 

//  check  if  left  and  right  subtree  is  BST 

return  is_BST( root->left,  min,  root->key  1)  &&  is_BST( root->right ,  root->key+1 ,  max) 

It  will  be  initially  called  as: 
is_BST(my_tree_root, KEY_MIN, KEY_MAX) 

Another  approach  will  be  to  do  inorder  traversal  of  the  Binary  tree.  If  the  inorder  traversal  produces  a  sorted 
sequence  of  keys  then  the  given  tree  is  a  BST.  To  check  if  the  inorder  sequence  is  sorted  remember  the  value  of 
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previously  visited  node  and  compare  it  against  the  current  node. 

Section  36.2:  If  a  given  input  tree  follows  Binary  search  tree 
property  or  not 

For  example 

if  the  input  is: 


Output  should  be  false: 

As  4  in  the  left  sub-tree  is  greater  than  the  root  value(3) 

If  the  input  is: 


Output  should  be  true 
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Chapter  37:  Binary  Tree  traversals 

Visiting  a  node  of  a  binary  tree  in  some  particular  order  is  called  traversals. 

Section  37.1:  Level  Order  traversal  -  Implementation 

For  example  if  the  given  tree  is: 


Level  order  traversal  will  be 

1234567 

Printing  node  data  level  by  level. 
Code: 

#include<iost ream> 
#include<queue> 
#include<malloc . h> 

using  namespace  std ; 

struct  node{ 

int  data; 
node  *left; 
node  *right; 

}; 


void  levelOrder(struct  node  *root){ 

if (root  ==  NULL)  return 

queue-mode  *>  Q 
Q.push( root)  ; 

while( ! Q. empty( ) ) { 
struct  node*  curr  =  Q.front(); 
cout<<  curr->data  <<" 

if (curr->left  !=  NULL)  Q. push(curr->  left); 

if (curr->right  !=  NULL)  Q.push(curr->  right); 

Q • pop ( )  ; 


} 
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} 

struct  node*  newNode(int  data) 

{ 

struct  node*  node  =  (struct  node*) 

malloc(sizeof (struct  node)); 

node->data  =  data; 
node->left  =  NULL; 
node->right  =  NULL; 


return(node) ; 


int  main(){ 


struct  node  *root 
root->lef t 
root->right 
root->lef t->left 
root->left->right 
root- >right-> left 


newNode(1 ) ; 
newNode(2) ; 
newNode(3) ; 
newNode(4) ; 
newNode(5) ; 
newNode(6) 


root->right->right  =  newNode(7) 


printf( "Level  Order  traversal  of  binary  tree  is  \n"); 
levelOrder( root) ; 

return  0; 


} 

Queue  data  structure  is  used  to  achieve  the  above  objective. 

Section  37.2:  Pre-order,  Inorder  and  Post  Order  traversal  of  a 
Binary  Tree 

Consider  the  Binary  Tree: 


Pre-order  traversal(root)  is  traversing  the  node  then  left  sub-tree  of  the  node  and  then  the  right  sub-tree  of  the 
node. 

So  the  pre-order  traversal  of  above  tree  will  be: 

1245367 

In-order  traversal(root)  is  traversing  the  left  sub-tree  of  the  node  then  the  node  and  then  right  sub-tree  of  the 
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node. 


So  the  in-order  traversal  of  above  tree  will  be: 

4251637 

Post-order  traversal(root)  is  traversing  the  left  sub-tree  of  the  node  then  the  right  sub-tree  and  then  the  node. 
So  the  post-order  traversal  of  above  tree  will  be: 

4526731 
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Chapter  38:  Lowest  common  ancestor  of  a 
Binary  Tree 

Lowest  common  ancestor  between  two  nodes  nl  and  n2  is  defined  as  the  lowest  node  in  the  tree  that  has  both  nl 
and  n2  as  descendants. 

Section  38.1:  Finding  lowest  common  ancestor 

Consider  the  tree: 


Lowest  common  ancestor  of  nodes  with  value  1  and  4  is  2 
Lowest  common  ancestor  of  nodes  with  value  1  and  5  is  3 
Lowest  common  ancestor  of  nodes  with  value  2  and  4  is  4 
Lowest  common  ancestor  of  nodes  with  value  1  and  2  is  2 
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Chapter  39:  Searching 

Section  39.1:  Binary  Search 

Introduction 

Binary  Search  is  a  Divide  and  Conquer  search  algorithm.  It  uses  0(  log  n)  time  to  find  the  location  of  an  element  in 
a  search  space  where  n  is  the  size  of  the  search  space. 

Binary  Search  works  by  halving  the  search  space  at  each  iteration  after  comparing  the  target  value  to  the  middle 
value  of  the  search  space. 

To  use  Binary  Search,  the  search  space  must  be  ordered  (sorted)  in  some  way.  Duplicate  entries  (ones  that 
compare  as  equal  according  to  the  comparison  function)  cannot  be  distinguished,  though  they  don't  violate  the 
Binary  Search  property. 

Conventionally,  we  use  less  than  (<)  as  the  comparison  function.  If  a  <  b,  it  will  return  true,  if  a  is  not  less  than  b  and 
b  is  not  less  than  a,  a  and  b  are  equal. 


Example  Question 

You  are  an  economist,  a  pretty  bad  one  though.  You  are  given  the  task  of  finding  the  equilibrium  price  (that  is,  the 
price  where  supply  =  demand)  for  rice. 

Remember  the  higher  a  price  is  set,  the  larger  the  supply  and  the  lesser  the  demand 

As  your  company  is  very  efficient  at  calculating  market  forces,  you  can  instantly  get  the  supply  and  demand  in  units 
of  rice  when  the  price  of  rice  is  set  at  a  certain  price  p. 

Your  boss  wants  the  equilibrium  price  ASAP,  but  tells  you  that  the  equilibrium  price  can  be  a  positive  integer  that  is 
at  most  1 0Al  7  and  there  is  guaranteed  to  be  exactly  1  positive  integer  solution  in  the  range.  So  get  going  with  your 
job  before  you  lose  it! 

You  are  allowed  to  call  functions  getSupply(k)  and  getDemand(k),  which  will  do  exactly  what  is  stated  in  the 
problem. 

Example  Explanation 

Here  our  search  space  is  from  1  to  1 0Al  7.  Thus  a  linear  search  is  infeasible. 

However,  notice  that  as  the  k  goes  up,  getSupply(k)  increases  and  getDemand(k)  decreases.  Thus,  for  any  x  >  y, 
getSupply(x)  -  getDemand(x)  >  getSupply(y)  -  getDemand (y).  Therefore,  this  search  space  is  monotonic  and 
we  can  use  Binary  Search. 

The  following  psuedocode  demonstrates  the  usage  of  Binary  Search: 


high  =  100000000000000000 
low  =  1 

while  high  -  low  >  1 

mid  =  (high  +  low)  /  2 
supply  =  getSupply(mid) 
demand  =  getDemand(mid) 
if  supply  >  demand 
high  =  mid 


<-  Upper  bound  of  search  space 
<-  Lower  bound  of  search  space 

<-  Take  the  middle  value 


<-  Solution  is  in  lower  half  of  search  space 
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else  if  demand  >  supply 
low  =  mid 


<-  Solution  is  in  upper  half  of  search  space 
<-  supply==demand  condition 
<-  Found  solution 


else 

return  mid 


This  algorithm  runs  in  ~0(log  10A17)  time.  This  can  be  generalized  to  ~0(log  S)  time  where  S  is  the  size  of  the 
search  space  since  at  every  iteration  of  the  while  loop,  we  halved  the  search  space  (from  [low:high]  to  either 
[low: mid]  or  [mid: high]). 

C  Implementation  of  Binary  Search  with  Recursion 

int  binsearch(int  a [ ’ ,  int  x,  int  low,  int  high)  { 
int  mid ; 

if  (low  >  high) 

return  -1  ; 

mid  =  (low  +  high)  /  2; 

if  (x  ==  a[mid] )  { 
return  (mid); 

}  else 

if  (x  <  a[mid])  { 

binsearch(a,  x,  low,  mid  -  1); 

}  else  { 

binsearch(a,  x,  mid  +  1,  high); 

} 


> 


Section  39.2:  Rabin  Karp 


The  Rabin-Karp  algorithm  or  Karp-Rabin  algorithm  is  a  string  searching  algorithm  that  uses  hashing  to  find  any  one 
of  a  set  of  pattern  strings  in  a  text.lts  average  and  best  case  running  time  is  0(n+m)  in  space  O(p),  but  its  worst-case 
time  is  O(nm)  where  n  is  the  length  of  the  text  and  m  is  the  length  of  the  pattern. 

Algorithm  implementation  in  java  for  string  matching 

void  Rabinf indPattern(String  text, String  pattern)! 

/* 

q  a  prime  number 
p  hash  value  for  pattern 
t  hash  value  for  text 

d  is  the  number  of  unique  characters  in  input  alphabet 

*/ 

int  d=1 28 ; 
int  q=1 00 ; 

int  n=text . length( )  ; 
int  m=pattern . length( )  ; 
int  t=0,p=0; 

int  h=1 ; 
int  i, j ; 

//hash  value  calculating  function 
for  (i=0;i<m-1 ;i++) 
h  =  (h*d)%q; 
for  (  i=0 ; i<m  i++) { 
p  =  (d*p  +  pattern . charAt (i) )%q ; 
t  =  (d*t  +  text . charAt ( i) )%q ; 

} 

//search  for  the  pattern 
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for(i=0;i<end-m  i++){ 
if ( p==t ) { 

//if  the  hash  value  matches  match  them  character  by  character 
for( j=0; j<m; j++) 

if (text . char At ( j+i) ! -pattern . char At (j ) ) 

break 

if(j==m  &&  i>=start) 

System. out . println( "Pattern  match  found  at  index  "+i) ; 

} 

if ( i<end-m) { 

t  =(d*(t  -  text .charAt(i)*h)  +  text .charAt(i+m) )%q ; 
if (t<0) 
t=t+q ; 

} 

} 

} 

While  calculating  hash  value  we  are  dividing  it  by  a  prime  number  in  order  to  avoid  collision.After  dividing  by  prime 
number  the  chances  of  collision  will  be  less,  but  still  ther  is  a  chance  that  the  hash  value  can  be  same  for  two 
strings, so  when  we  get  a  match  we  have  to  check  it  character  by  character  to  make  sure  that  we  got  a  proper 
match. 

t  =(d*(t  -  text.charAt(i)*h)  +  text.charAt(i+m))%q; 

This  is  to  recalculate  the  hash  value  for  pattern, first  by  removing  the  left  most  character  and  then  adding  the  new 
character  from  the  text. 

Section  39.3:  Analysis  of  Linear  search  (Worst,  Average  and 
Best  Cases) 

We  can  have  three  cases  to  analyze  an  algorithm: 

1 .  Worst  Case 

2.  Average  Case 

3.  Best  Case 


#include  <stdio.h> 

//  Linearly  search  x  in  arr[].  If  x  is  present  then  return  the  index, 

//  otherwise  return  -1 

int  search(int  arr[],  int  n,  int  x) 

{ 

int  i 

for  (i=0;  i<n  i++) 

{ 

if  (arr[i]  ==  x) 
return  i 

} 

return  -1 ; 

} 

/*  Driver  program  to  test  above  functions*/ 
int  main() 
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{ 


int  arr[]  =  {1,  10,  30,  15}; 
int  x  =  30; 

int  n  =  sizeof (arr)/sizeof(arr[0] ) ; 

printf("%d  is  present  at  index  %d",  x,  search(arr,  n,  x)); 

getchar ( ) ; 

return  0; 


Worst  Case  Analysis  (Usually  Done) 

In  the  worst  case  analysis,  we  calculate  upper  bound  on  running  time  of  an  algorithm.  We  must  know  the  case  that 
causes  maximum  number  of  operations  to  be  executed.  For  Linear  Search,  the  worst  case  happens  when  the 
element  to  be  searched  (x  in  the  above  code)  is  not  present  in  the  array.  When  x  is  not  present,  the  search() 
functions  compares  it  with  all  the  elements  of  arr[]  one  by  one.  Therefore,  the  worst  case  time  complexity  of  linear 
search  would  be  0(n) 

Average  Case  Analysis  (Sometimes  done) 

In  average  case  analysis,  we  take  all  possible  inputs  and  calculate  computing  time  for  all  of  the  inputs.  Sum  all  the 
calculated  values  and  divide  the  sum  by  total  number  of  inputs.  We  must  know  (or  predict)  distribution  of  cases.  For 
the  linear  search  problem,  let  us  assume  that  all  cases  are  uniformly  distributed  (including  the  case  of  x  not  being 
present  in  array).  So  we  sum  all  the  cases  and  divide  the  sum  by  (n+1 ).  Following  is  the  value  of  average  case  time 
complexity. 


Ey  aw 

(n+TJ 

Average  Case  Time  = 

0({n+l)*(n+2)/2) 

(n-fTJ 


=  0(n) 


Best  Case  Analysis  (Bogus) 

In  the  best  case  analysis,  we  calculate  lower  bound  on  running  time  of  an  algorithm.  We  must  know  the  case  that 
causes  minimum  number  of  operations  to  be  executed.  In  the  linear  search  problem,  the  best  case  occurs  when  x 
is  present  at  the  first  location.  The  number  of  operations  in  the  best  case  is  constant  (not  dependent  on  n).  So  time 
complexity  in  the  best  case  would  be  0(1 )  Most  of  the  times,  we  do  worst  case  analysis  to  analyze  algorithms.  In  the 
worst  analysis,  we  guarantee  an  upper  bound  on  the  running  time  of  an  algorithm  which  is  good  information.  The 
average  case  analysis  is  not  easy  to  do  in  most  of  the  practical  cases  and  it  is  rarely  done.  In  the  average  case 
analysis,  we  must  know  (or  predict)  the  mathematical  distribution  of  all  possible  inputs.  The  Best  Case  analysis  is 
bogus.  Guaranteeing  a  lower  bound  on  an  algorithm  doesn't  provide  any  information  as  in  the  worst  case,  an 
algorithm  may  take  years  to  run. 

For  some  algorithms,  all  the  cases  are  asymptotically  same,  i.e.,  there  are  no  worst  and  best  cases.  For  example, 
Merge  Sort.  Merge  Sort  does  0(nLogn)  operations  in  all  cases.  Most  of  the  other  sorting  algorithms  have  worst  and 
best  cases.  For  example,  in  the  typical  implementation  of  Quick  Sort  (where  pivot  is  chosen  as  a  corner  element), 
the  worst  occurs  when  the  input  array  is  already  sorted  and  the  best  occur  when  the  pivot  elements  always  divide 
array  in  two  halves.  For  insertion  sort,  the  worst  case  occurs  when  the  array  is  reverse  sorted  and  the  best  case 
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occurs  when  the  array  is  sorted  in  the  same  order  as  output. 

Section  39.4:  Binary  Search:  On  Sorted  Numbers 

It's  easiest  to  show  a  binary  search  on  numbers  using  pseudo-code 

int  array[1000]  =  {  sorted  list  of  numbers  }; 

int  N  =  100;  //  number  of  entries  in  search  space; 

int  high,  low,  mid;  //  our  temporaries 
int  x;  //  value  to  search  for 

low  =  0  ; 
high  =  N  -1  ; 
while(low  <  high) 

{ 

mid  =  (low  +  h i g  h ) / 2 ; 
if (array[mid]  <  x) 
low  =  mid  +  1  ; 

else 

high  =  mid; 

} 

if (array! low ]  ==  x) 

//  found,  index  is  low 

else 

//  not  found 

Do  not  attempt  to  return  early  by  comparing  array[mid]  to  x  for  equality.  The  extra  comparison  can  only  slow  the 
code  down.  Note  you  need  to  add  one  to  low  to  avoid  becoming  trapped  by  integer  division  always  rounding  down. 

Interestingly,  the  above  version  of  binary  search  allows  you  to  find  the  smallest  occurrence  of  x  in  the  array.  If  the 
array  contains  duplicates  of  x,  the  algorithm  can  be  modified  slightly  in  order  for  it  to  return  the  largest  occurrence 
of  x  by  simply  adding  to  the  if  conditional: 

while(low  <  high) 

{ 

mid  =  low  +  ((high  low)  /  2); 

if (array[mid]  <  x  ||  (array[mid]  ==  x  &&  array[mid  +  1]  ==  x)) 
low  =  mid  +  1 ; 

else 

high  =  mid; 

} 

Note  that  instead  of  doing  mid  =  (low  +  high)  /  2,  it  may  also  be  a  good  idea  to  try  mid  =  low  +  ((high  low) 
/  2)  for  implementations  such  as  Java  implementations  to  lower  the  risk  of  getting  an  overflow  for  really  large 
inputs. 

Section  39.5:  Linear  search 

Linear  search  is  a  simple  algorithm.  It  loops  through  items  until  the  query  has  been  found,  which  makes  it  a  linear 
algorithm  -  the  complexity  is  O(n),  where  n  is  the  number  of  items  to  go  through. 

Why  O(n)?  In  worst-case  scenario,  you  have  to  go  through  all  of  the  n  items. 

It  can  be  compared  to  looking  for  a  book  in  a  stack  of  books  -  you  go  through  them  all  until  you  find  the  one  that 
you  want. 

Below  is  a  Python  implementation: 
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def  linear_search(searchable_list,  query) : 
for  x  in  searchable_list : 
if  query  ==  x : 

return  True 
return  False 

linear_search( [' apple ' ,  'banana',  'carrot',  'fig',  'garlic'',  'fig')  #returns  True 


GoalKicker.com  -  Algorithms  Notes  for  Professionals 


176 


Chapter  40:  Substring  Search 

Section  40.1:  Introduction  To  Knuth-Morris-Pratt  (KMP) 
Algorithm 

Suppose  that  we  have  a  text  and  a  pattern.  We  need  to  determine  if  the  pattern  exists  in  the  text  or  not.  For 
example: 

+ - + —  + — + — + —  + — + — + —  + — + 

|  Index  |0|1|2|3|4|5|6|7| 

+ - + —  +  —  +  — + —  +  —  +  — + —  +  —  + 

|  Text  |a|b|c|b|c|g|l|x| 

+ - + —  +  —  +  — + —  +  —  +  — + —  +  —  + 

+ - + — + — + — + — + 

|  Index  |  0  |  1  |  2  |  3  | 

+ - + — + — + — + — + 

|  Pattern  |  b  |  c  |  g  |  l  | 

+ - +  —  +  — + — + —  + 

This  pattern  does  exist  in  the  text.  So  our  substring  search  should  return  3,  the  index  of  the  position  from  which  this 
pattern  starts.  So  how  does  our  brute  force  substring  search  procedure  work? 

What  we  usually  do  is:  we  start  from  the  Oth  index  of  the  text  and  the  0th  index  of  our  *pattern  and  we  compare 
Text[0]  with  Pattern[0].  Since  they  are  not  a  match,  we  go  to  the  next  index  of  our  text  and  we  compare  Text[1] 
with  Pattern[0].  Since  this  is  a  match,  we  increment  the  index  of  out  pattern  and  the  index  of  the  Text  also.  We 
compare  Text[2]  with  Pattern[1],  They  are  also  a  match.  Following  the  same  procedure  stated  before,  we  now 
compare  Text[3]  with  Pattern[2],  As  they  do  not  match,  we  start  from  the  next  position  where  we  started  finding 
the  match.  That  is  index  2  of  the  Text.  We  compare  Text[2]  with  Pattern[0].  They  don't  match.  Then  incrementing 
index  of  the  Text,  we  compare  Text[3]  with  Pattern[0].  They  match.  Again  Text[4]  and  Pattern[1]  match,  Text[5] 
and  Pattern[2]  match  and  Text[6]  and  Pattern[3]  match.  Since  we've  reached  the  end  of  our  Pattern,  we  now 
return  the  index  from  which  our  match  started,  that  is  3.  If  our  pattern  was:  bcgll,  that  means  if  the  pattern  didn't 
exist  in  our  text,  our  search  should  return  exception  or  -1  or  any  other  predefined  value.  We  can  clearly  see  that,  in 
the  worst  case,  this  algorithm  would  take  0(mn)  time  where  m  is  the  length  of  the  Text  and  n  is  the  length  of  the 
Pattern.  Flow  do  we  reduce  this  time  complexity?  This  is  where  KMP  Substring  Search  Algorithm  comes  into  the 
picture. 

The  Knuth-Morris-Pratt  String  Searching  Algorithm  or  KMP  Algorithm  searches  for  occurrences  of  a  "Pattern"  within 
a  main  "Text"  by  employing  the  observation  that  when  a  mismatch  occurs,  the  word  itself  embodies  sufficient 
information  to  determine  where  the  next  match  could  begin,  thus  bypassing  re-examination  of  previously  matched 
characters.  The  algorithm  was  conceived  in  1 970  by  Donuld  Knuth  and  Vaughan  Pratt  and  independently  by  ames 
FI.  Morris.  The  trio  published  it  jointly  in  1977. 

Let's  extend  our  example  Text  and  Pattern  for  better  understanding: 

|  Index  |0  |1  | 2  |3  |4  |5  |6  |7  |8  |9  | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 

+ - 

|  T ext  |a  |b  |c  |x  |a  |b  |c  |d  |a  |b  |x  |a  |b  |c  |d  |a  |b  |c  |d  |a  |b  |c  |y  | 

+ - +  —  +  — + — + —  +  —  +  — + —  +  —  + 

|  Index  |0|1|2|3|4|5|6|7| 
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+ - +  —  +  —  +  —  +  —  +  —  +  —  +  —  +  —  + 

|  Pattern  |a|b|c|d|a|b|c|y| 
+ - +  —  +  —  +  — + —  +  —  +  — + —  +  —  + 


At  first,  our  Text  and  Pattern  matches  till  index  2.  Text[3]  and  Pattern[3]  doesn't  match.  So  our  aim  is  to  not  go 
backwards  in  this  Text,  that  is,  in  case  of  a  mismatch,  we  don't  want  our  matching  to  begin  again  from  the  position 
that  we  started  matching  with.  To  achieve  that,  we'll  look  for  a  suffix  in  our  Pattern  right  before  our  mismatch 
occurred  (substring  abc),  which  is  also  a  prefix  of  the  substring  of  our  Pattern.  For  our  example,  since  all  the 
characters  are  unique,  there  is  no  suffix,  that  is  the  prefix  of  our  matched  substring.  So  what  that  means  is,  our 
next  comparison  will  start  from  index  0.  Hold  on  for  a  bit,  you'll  understand  why  we  did  this.  Next,  we  compare 
Text[3]  with  Pattern[0]  and  it  doesn't  match.  After  that,  for  Text  from  index  4  to  index  9  and  for  Pattern  from  index 
0  to  index  5,  we  find  a  match.  We  find  a  mismatch  in  Text[10]  and  Pattern[6],  So  we  take  the  substring  from  Pattern 
right  before  the  point  where  mismatch  occurs  (substring  abcdabc),  we  check  for  a  suffix,  that  is  also  a  prefix  of  this 
substring.  We  can  see  here  ab  is  both  the  suffix  and  prefix  of  this  substring.  What  that  means  is,  since  we've 
matched  until  Text[10],  the  characters  right  before  the  mismatch  is  ab.  What  we  can  infer  from  it  is  that  since  ab  is 
also  a  prefix  of  the  substring  we  took,  we  don't  have  to  check  ab  again  and  the  next  check  can  start  from  Text[10] 
and  Pattern[2],  We  didn't  have  to  look  back  to  the  whole  Text,  we  can  start  directly  from  where  our  mismatch 
occurred.  Now  we  check  Text[10]  and  Pattern[2],  since  it's  a  mismatch,  and  the  substring  before  mismatch  (abc) 
doesn't  contain  a  suffix  which  is  also  a  prefix,  we  check  Text[10]  and  Pattern[0],  they  don't  match.  After  that  for 
Text  from  index  11  to  index  17  and  for  Pattern  from  index  0  to  index  6.  We  find  a  mismatch  in  Text[18]  and 
Pattern[7],  So  again  we  check  the  substring  before  mismatch  (substring  abcdabc)  and  find  abc  is  both  the  suffix 
and  the  prefix.  So  since  we  matched  till  Pattern[7],  abc  must  be  before  Text[18],  That  means,  we  don't  need  to 
compare  until  Text[17]  and  our  comparison  will  start  from  Text[18]  and  Pattern[3],  Thus  we  will  find  a  match  and 
we'll  return  15  which  is  our  starting  index  of  the  match.  This  is  how  our  KMP  Substring  Search  works  using  suffix 
and  prefix  information. 

Now,  how  do  we  efficiently  compute  if  suffix  is  same  as  prefix  and  at  what  point  to  start  the  check  if  there  is  a 
mismatch  of  character  between  Text  and  Pattern.  Let's  take  a  look  at  an  example: 

+ - + — + — + — + — + — + — + — + — + 

|  Index  |0|1|2|3|4|5|6|7| 

+ - + — + — + — + — + — + — + — + — + 

|  Pattern  |a|b|c|d|a|b|c|a| 

+ - +  —  +  — + — + —  +  —  +  — + —  +  —  + 


We'll  generate  an  array  containing  the  required  information.  Let's  call  the  array  S.  The  size  of  the  array  will  be  same 
as  the  length  of  the  pattern.  Since  the  first  letter  of  the  Pattern  can't  be  the  suffix  of  any  prefix,  we'll  put  S[0]  =  0.  We 
take  i  =  1  and  j  =  0  at  first.  At  each  step  we  compare  Pattern[i]  and  Pattern[j]  and  increment  i.  If  there  is  a  match 
we  put  S[i]  =  j  +  1  and  increment  j,  if  there  is  a  mismatch,  we  check  the  previous  value  position  of  j  (if  available)  and 
set  j  =  S[j-1]  (if  j  is  not  equal  to  0),  we  keep  doing  this  until  S|j]  doesn't  match  with  S[i]  or  j  doesn't  become  0.  For  the 
later  one,  we  put  S[i]  =  0.  For  our  example: 

j  i 

+ - + — + — + — + — + — + — + — + — + 

|  Index  |0|1|2|3|4|5|6|7| 

+ - +  —  +  —  +  —  +  —  +  —  +  —  +  —  +  —  + 

|  Pattern  |a|b|c|d|a|b|c|a| 

+ - +  —  +  — + — + —  +  —  +  — + —  +  —  + 


Patternlj]  and  Pattern[i]  don't  match,  so  we  increment  i  and  since  j  is  0,  we  don't  check  the  previous  value  and  put 
Pattern[i]  =  0.  If  we  keep  incrementing  i,  for  i  =  4,  we'll  get  a  match,  so  we  put  S[i]  =  S[4]  =j  +  1=  0  +  1  =  1  and 
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increment  j  and  i.  Our  array  will  look  like: 


j  i 

+ - +  —  +  —  +  — + —  +  —  +  — + —  +  —  + 

|  Index  |0|1|2|3|4|5|6|7| 

+ - +  —  +  —  +  —  +  —  +  —  +  —  +  —  +  —  + 

|  Pattern  |a|b|c|d|a|b|c|a| 

+ - + — + — + — + — + — + — + — + — + 

I  S  |  0  |  0  |  0  |  0  |  1  |  |  |  | 

+ - + —  + —  + —  +  —  + —  + —  + —  + —  + 

Since  Pattern[1]  and  Pattern[5]  is  a  match,  we  put  S[i]  =  S[5]  =  j  +  1  =  1  +  1  =  2.  If  we  continue,  we'll  find  a 
mismatch  for  j  =  3  and  i  =  7.  Since  j  is  not  equal  to  0,  we  put  j  =  S[j-1].  And  we'll  compare  the  characters  at  i  and  j 
are  same  or  not,  since  they  are  same,  we'll  put  S[i]  =  j  +  1 .  Our  completed  array  will  look  like: 

+ - +  —  +  — + — + —  +  —  +  — + —  +  —  + 

I  S  I  0  I  0  I  0  I  0  I  1  I  2  I  3  I  1  I 
+ - + —  + —  + —  +  —  +  —  + —  +  —  +  —  + 

This  is  our  required  array.  Here  a  nonzero-value  of  S[i]  means  there  is  a  S[i]  length  suffix  same  as  the  prefix  in  that 
substring  (substring  from  0  to  i)  and  the  next  comparison  will  start  from  S[i]  +  1  position  of  the  Pattern.  Our 
algorithm  to  generate  the  array  would  look  like: 

Procedure  GenerateSuf f ixArray (Pattern) : 

i  :=  1 
j  :=  0 

n  :=  Pattern . length 
while  i  is  less  than  n 

if  Pattern[i  is  equal  to  Pattern[j] 

S[i]  :=  j  +  1 
j  :=  j  +  1 
i  :=  i  +  1 

else 

if  j  is  not  equal  to  0 

1  :=  S[j-1] 

else 

S[i]  :=  0 

i  :=  i  +  1 

end  if 
end  if 

end  while 

The  time  complexity  to  build  this  array  is  0(n)  and  the  space  complexity  is  also  0(n).  To  make  sure  if  you  have 
completely  understood  the  algorithm,  try  to  generate  an  array  for  pattern  aabaabaa  and  check  if  the  result  matches 
with  this  one. 

Now  let's  do  a  substring  search  using  the  following  example: 

+ - + — + — + — + — + — + — + — + — + — + —  + — + —  + 

|  Index  |0|1|2|3|4|5|6|7|8|9|10|11| 

+ - + — + — + — + — + — + — + — + — +  — + —  + — +  —  + 

|  Text  |a|b|x|a|b|c|a|b|c|a|b|y| 

+ - +  —  +  — + — + —  +  —  +  — + —  +  —  +  — + —  +  —  +  —  + 

+ - +  —  +  — + — + —  +  —  +  —  + 

|  Index  |0|1|2|3|4|5| 
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+ - +  —  +  —  +  —  + —  +  —  +  —  + 

|  Pattern  |a|b|c|a|b|y| 
+ - +  —  +  — + — + —  +  —  +  —  + 

|  S  I  0  I  0  I  0  I  1  I  2  I  0  I 

+ - + - + - + - + - + - + - + 


We  have  a  Text,  a  Pattern  and  a  pre-calculated  array  S  using  our  logic  defined  before.  We  compare  Text[0]  and 
Pattern[0]  and  they  are  same.  Text[1]  and  Pattern[1]  are  same.  Text[2]  and  Pattern[2]  are  not  same.  We  check 
the  value  at  the  position  right  before  the  mismatch.  Since  S[1]  is  0,  there  is  no  suffix  that  is  same  as  the  prefix  in  our 
substring  and  our  comparison  starts  at  position  S[1],  which  is  0.  So  Pattern[0]  is  not  same  as  Text[2],  so  we  move 
on.  Text[3]  is  same  as  Pattern[0]  and  there  is  a  match  till  Text[8]  and  Pattern[5],  We  go  one  step  back  in  the  S 
array  and  find  2.  So  this  means  there  is  a  prefix  of  length  2  which  is  also  the  suffix  of  this  substring  (abcab)  which  is 
ab.  That  also  means  that  there  is  an  ab  before  Text[8],  So  we  can  safely  ignore  Pattern[0]  and  Pattern[1]  and  start 
our  next  comparison  from  Pattern[2]  and  Text[8],  If  we  continue,  we'll  find  the  Pattern  in  the  Text.  Our  procedure 
will  look  like: 

Procedure  KMP(Text,  Pattern) 

GenerateSuf f ixAr ray (Pattern) 
m  :=  Text. Length 
n  :=  Pattern . Length 

i  :=  0 
j  :=  0 

while  i  is  less  than  m 

if  Pattern)]]  is  equal  to  Text[i] 
j  :=  j  +  1 
i  :=  i  +  1 

if  j  is  equal  to  n 

Return  (j-i) 

else  if  i  <  m  and  Pattern)]]  is  not  equal  t  Text[i 
if  j  is  not  equal  to  0 
j  =  S[j-1] 

else 

i  :=  i  +  1 

end  if 
end  if 
end  while 
Return  -1 

The  time  complexity  of  this  algorithm  apart  from  the  Suffix  Array  Calculation  is  0(m).  Since  GenerateSuffixArray  takes 
0(n),  the  total  time  complexity  of  KMP  Algorithm  is:  0(m+n). 

PS:  If  you  want  to  find  multiple  occurrences  of  Pattern  in  the  Text,  instead  of  returning  the  value,  print  it/store  it  and 
set  j  :=  S[  j-l  ].  Also  keep  a  flag  to  track  whether  you  have  found  any  occurrence  or  not  and  handle  it 
accordingly. 

Section  40.2:  Introduction  to  Rabin-Karp  Algorithm 

Rabin-Karp  Algorithm  is  a  string  searching  algorithm  created  by  Richard  M.  Karp  and  Michael  O.  Rabin  that  uses 
hashing  to  find  any  one  of  a  set  of  pattern  strings  in  a  text. 

A  substring  of  a  string  is  another  string  that  occurs  in.  For  example,  ver  is  a  substring  of  stackoverflow.  Not  to  be 
confused  with  subsequence  because  cover  is  a  subsequence  of  the  same  string.  In  other  words,  any  subset  of 
consecutive  letters  in  a  string  is  a  substring  of  the  given  string. 

In  Rabin-Karp  algorithm,  we'll  generate  a  hash  of  our  pattern  that  we  are  looking  for  &  check  if  the  rolling  hash  of 
our  text  matches  the  pattern  or  not.  If  it  doesn't  match,  we  can  guarantee  that  the  pattern  doesn't  exist  in  the  text. 
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However,  if  it  does  match,  the  pattern  can  be  present  in  the  text.  Let's  look  at  an  example: 

Let's  say  we  have  a  text:  yeminsajid  and  we  want  to  find  out  if  the  pattern  nsa  exists  in  the  text.  To  calculate  the 
hash  and  rolling  hash,  we'll  need  to  use  a  prime  number.  This  can  be  any  prime  number.  Let's  take  prime  =  11  for 
this  example.  We'll  determine  hash  value  using  this  formula: 

(1st  letter)  X  (prime)  +  (2nd  letter)  X  (prime)1  +  (3rd  letter)  X  (prime)2  X  +  . 

We'll  denote: 


a 

-> 

i 

g 

-> 

7 

m 

-> 

13 

s 

-> 

19 

y 

LO 

CM 

A 

1 

b 

-> 

2 

h 

-> 

8 

n 

-> 

14 

t 

-> 

20 

z 

->  26 

c 

-> 

3 

i 

-> 

9 

0 

-> 

15 

u 

-> 

21 

d 

-> 

4 

1 

-> 

10 

P 

-> 

16 

V 

-> 

22 

e 

-> 

5 

k 

-> 

11 

q 

-> 

17 

w 

-> 

23 

f 

-> 

6 

1 

-> 

12 

r 

-> 

18 

X 

-> 

24 

The  hash  value  of  nsa  will  be: 

14  X  11°  +  19  X  II1  +  1  X  II2  =  344 

Now  we  find  the  rolling-hash  of  our  text.  If  the  rolling  hash  matches  with  the  hash  value  of  our  pattern,  we'll  check  if 
the  strings  match  or  not.  Since  our  pattern  has  3  letters,  we'll  take  1  st  3  letters  yem  from  our  text  and  calculate 
hash  value.  We  get: 

25  X  11°  +  5  X  II1  +  13  X  II2  =  1653 

This  value  doesn't  match  with  our  pattern's  hash  value.  So  the  string  doesn't  exists  here.  Now  we  need  to  consider 
the  next  step.  To  calculate  the  hash  value  of  our  next  string  emi.  We  can  calculate  this  using  our  formula.  But  that 
would  be  rather  trivial  and  cost  us  more.  Instead,  we  use  another  technique. 

•  We  subtract  the  value  of  the  First  Letter  of  Previous  String  from  our  current  hash  value.  In  this  case,  y.  We 
get,  1653  -  25  =  1628. 

•  We  divide  the  difference  with  our  prime,  which  is  11  for  this  example.  We  get,  1 628  /  11  =  148. 

•  We  add  new  letter  X  (prime)D-1,  where  m  is  the  length  of  the  pattern,  with  the  quotient,  which  is  i  =  9.  We 
get,  148  +  9  X  II2  =  1237. 

The  new  hash  value  is  not  equal  to  our  patterns  hash  value.  Moving  on,  for  n  we  get: 

Previous  String:  emi 

First  Letter  of  Previous  String:  e(5) 

New  Letter:  n(14) 

New  String:  "min" 

1237  -  5  =  1232 
1232  /  11  =  112 
112  +  14  X  II2  =  1806 

It  doesn't  match.  After  that,  for  s,  we  get: 

Previous  String:  min 

First  Letter  of  Previous  String:  m(13) 

New  Letter:  s(19) 

New  String:  "ins" 

1806  -  13  =  1793 
1793  /  11  =  163 
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163  +  19  X  11 


2462 


It  doesn't  match.  Next,  for  a,  we  get: 

Previous  String:  ins 

First  Letter  of  Previous  String:  i(9) 

New  Letter :  a(1 ) 

New  String:  "nsa" 

2462  -  9  =  2453 
2453  /  11  =  223 
223+1  X  II2  =  344 

It's  a  match!  Now  we  compare  our  pattern  with  the  current  string.  Since  both  the  strings  match,  the  substring  exists 
in  this  string.  And  we  return  the  starting  position  of  our  substring. 

The  pseudo-code  will  be: 

Hash  Calculation: 

Procedure  Calculate  Hash(String,  Prime,  x) : 

hash  :=  0  //  Here  x  denotes  the  length  to  be  considered 

for  m  from  1  to  x  //  to  find  the  hash  value 

hash  :=  hash  +  (Value  of  String [m] )  CL1 

end  for 
Return  hash 

Hash  Recalculation: 

Procedure  Recalculate  Hash(String,  Curr,  Prime,  Hash)  : 

Hash  :=  Hash  Value  of  String[Curr]  //here  Curr  denotes  First  Letter  of  Previous  String 
Hash  :=  Hash  /  Prime 
m  :=  String . length 
New  :=  Curr  +  m  -  1 

Hash  :=  Hash  +  (Value  of  String [New] )  CL1 
Return  Hash 

String  Match: 

Procedure  String  Match(Text,  Pattern,  m) : 
for  i  f  rom  m  to  Pattern  length  +  m  -  1 
if  Text[i  is  not  equal  to  Pattern[i 

Return  false 
end  if 
end  for 
Return  true 

Rabin-Karp: 

Procedure  Rabin  Karp(Text,  Pattern,  Prime): 
m  :=  Pattern . Length 

HashValue  :=  Calculate  Hash(Pattern,  Prime,  m) 

CurrValue  :=  Calculate  Hash(Pattern,  Prime,  m) 
for  i  f  rom  1  to  Text. length  -  m 

if  HashValue  ==  CurrValue  and  String  Match(Text,  Pattern,  i)  is  true 
Return  i 
end  if 

CurrValue  :=  Recalculate  Hash(String,  i+1 ,  Prime,  CurrValue) 

end  for 
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Return  -1 


If  the  algorithm  doesn't  find  any  match,  it  simply  returns  -1. 

This  algorithm  is  used  in  detecting  plagiarism.  Given  source  material,  the  algorithm  can  rapidly  search  through  a 
paper  for  instances  of  sentences  from  the  source  material,  ignoring  details  such  as  case  and  punctuation.  Because 
of  the  abundance  of  the  sought  strings,  single-string  searching  algorithms  are  impractical  here.  Again,  Knuth- 
Morris-Pratt  algorithm  or  Boyer-Moore  String  Search  algorithm  is  faster  single  pattern  string  searching 
algorithm,  than  Rabin-Karp.  However,  it  is  an  algorithm  of  choice  for  multiple  pattern  search.  If  we  want  to  find  any 
of  the  large  number,  say  k,  fixed  length  patterns  in  a  text,  we  can  create  a  simple  variant  of  the  Rabin-Karp 
algorithm. 

For  text  of  length  n  and  p  patterns  of  combined  length  m,  its  average  and  best  case  running  time  is  0(n+m)  in 
space  O(p),  but  its  worst-case  time  is  O(nm). 

Section  40.3:  Python  Implementation  of  KMP  algorithm 

Haystack:  The  string  in  which  given  pattern  needs  to  be  searched. 

Needle:  The  pattern  to  be  searched. 

Time  complexity:  Search  portion  (strstr  method)  has  the  complexity  0(n)  where  n  is  the  length  of  haystack  but  as 
needle  is  also  pre  parsed  for  building  prefix  table  0(m)  is  required  for  building  prefix  table  where  m  is  the  length  of 
the  needle. 

Therefore,  overall  time  complexity  for  KMP  is  0(n+m) 

Space  complexity:  0(m)  because  of  prefix  table  on  needle. 

Note:  Following  implementation  returns  the  start  position  of  match  in  haystack  (if  there  is  a  match)  else  returns  -1, 
for  edge  cases  like  if  needle/haystack  is  an  empty  string  or  needle  is  not  found  in  haystack. 

def  get_prefix_table(needle) : 
prefix_set  =  set() 
n  =  len(needle) 
prefix_table  =  [0]*n 
delimeter  =  1 
while(delimeter<n) : 

prefix_set.add(needle[ : delimeter] ) 

1  =  1 

while( j<delimeter+1 ) : 

if  needle[j  :delimeter+1 ]  in  prefix_set 

prefix_table[delimeter]  =  delimeter  -  j  +  1 
break 

j  +=  1 

delimeter  +=  1 
return  prefix_table 

def  strstr(haystack,  needle): 

#  m:  denoting  the  position  within  S  where  the  prospective  match  for  W  begins 

#  i  denoting  the  index  of  the  currently  considered  character  in  W. 
haystack_len  =  len(haystack) 

needle_len  =  len(needle) 

if  (needle_len  >  haystack_len)  or  (not  haystack_len)  or  (not  needle_len) : 

return  -1 

prefix_table  =  get_pref ix_table(needle) 

m  =  i  =  0 

while( (i<needle_len)  and  (m<haystack_len) ) : 
if  haystack[m]  ==  needle[i] 

i  +=  1 
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m  +=  1 

else 

if  i  ! =  0  : 

i  =  prefix_table[ i  1 ] 

else 

m  +=  1 

if  i==needle_len  and  haystack [m-1 ]  ==  needle[i-1]: 
return  m  -  needle_len 

else 

return  -1 

if _ name _ ==  '__main _ '  : 

needle  =  'abcaby' 
haystack  =  ' abxabcabcaby ' 
print  strstr(haystack,  needle) 

Section  40.4:  KMP  Algorithm  in  C 

Given  a  text  txt  and  a  pattern  pat,  the  objective  of  this  program  will  be  to  print  all  the  occurance  of  pat  in  txt. 

Examples: 

Input: 

txt [ ]  =  "THIS  IS  A  TEST  TEXT" 
pat [ ]  =  "TEST" 

output: 

Pattern  found  at  index  10 

Input: 

txt [ ]  =  " AAB AACAADAABAAABAA " 

pat [ ]  =  " AABA" 

output: 

Pattern  found  at  index  0 
Pattern  found  at  index  9 
Pattern  found  at  index  13 

C  Language  Implementation: 

//  C  program  for  implementation  of  KMP  pattern  searching 
//  algorithm 

#include<stdio . h> 

#include<string .h> 

#include<stdlib . h> 

void  computeLPSArray(char  *pat,  int  M,  int  *lps); 

void  KMPSearch(char  *pat,  char  *txt) 

{ 

int  M  =  strlen(pat)  ; 
int  N  =  strlen(txt); 

//  create  lps[]  that  will  hold  the  longest  prefix  suffix 
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//  values  for  pattern 

int  *lps  =  (int  *)malloc(sizeof (int)*M) ; 
int  j  =0;  //  index  for  pat[] 

//  Preprocess  the  pattern  (calculate  lps[]  array) 
computeLPSArray (pat ,  M,  lps) ; 

int  i  =  0;  //  index  for  txt[] 

while  (i  <  N) 

{ 

if  (pat [ j  ]  ==  txt [ i  ] ) 

{ 

j++; 

i++; 

} 

if  (j  ==  M) 

{ 

printf( "Found  pattern  at  index  %d  \n",  i-j); 
j  =  lps [ j - 1  ]  ; 

} 

//  mismatch  after  j  matches 

else  if  (i  <  N  &&  pat[j]  !=  txt[i]) 

{ 

//  Do  not  match  lps[0. .lps[j-1  ] ]  characters, 
//  they  will  match  anyway 
if  (j  ! =  0) 
j  =  lps [ j - 1  ]  ; 

else 

i  =  i+1  ; 

} 

} 

free(lps);  //  to  avoid  memory  leak 


void  computeLPSArray(char  *pat,  int  M,  int  *lps) 

{ 

int  len  =0;  //  length  of  the  previous  longest  prefix  suffix 

int  i 

lps [ 0 ]  =  0  ;  //  lps[0]  is  always  0 
i  =  1  ; 

//  the  loop  calculates  lps[i]  for  i  =  1  to  M-l 

while  (i  <  M) 

{ 

if  ( pat [ i ]  ==  pat[len]) 

{ 

len++; 

lps[i]  =  len 

i++; 

} 

else  //  (pat[i]  !=  pat [len]) 

{ 

if  (len  !=  0) 

{ 

//  This  is  tricky.  Consider  the  example 
//  AAACAAAA  and  i  =  7. 
len  =  lps[ len-1  ]  ; 

//  Also,  note  that  we  do  not  increment  i  here 
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} 

else  //  if  (len  ==  0) 

{ 

lps[i]  =  0; 
i++; 

} 

} 

} 

} 

//  Driver  program  to  test  above  function 
int  main() 

{ 

char  *txt  =  "ABABDABACDABABCABAB" ; 
char  *pat  =  "ABABCABAB"; 

KMPSea rch ( pat ,  txt )  ; 

return  0; 

} 

Output: 

Found  pattern  at  index  10 

Reference: 

http://www.geeksforgeeks.org/searching-for-patterns-set-2-kmp-algorithm/ 
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Chapter  41:  Breadth-First  Search 

Section  41.1:  Finding  the  Shortest  Path  from  Source  to  other 
Nodes 

Breadth-first-search  (BFS)  is  an  algorithm  for  traversing  or  searching  tree  or  graph  data  structures.  It  starts  at  the 
tree  root  (or  some  arbitrary  node  of  a  graph,  sometimes  referred  to  as  a  'search  key')  and  explores  the  neighbor 
nodes  first,  before  moving  to  the  next  level  neighbors.  BFS  was  invented  in  the  late  1 950s  by  Edward  Forrest  Moore. 
who  used  it  to  find  the  shortest  path  out  of  a  maze  and  discovered  independently  by  C.  Y.  Lee  as  a  wire  routing 
algorithm  in  1 961 . 

The  processes  of  BFS  algorithm  works  under  these  assumptions: 

1 .  We  won't  traverse  any  node  more  than  once. 

2.  Source  node  or  the  node  that  we're  starting  from  is  situated  in  level  0. 

3.  The  nodes  we  can  directly  reach  from  source  node  are  level  1  nodes,  the  nodes  we  can  directly  reach  from 
level  1  nodes  are  level  2  nodes  and  so  on. 

4.  The  level  denotes  the  distance  of  the  shortest  path  from  the  source. 

Let's  see  an  example: 


Let's  assume  this  graph  represents  connection  between  multiple  cities,  where  each  node  denotes  a  city  and  an 
edge  between  two  nodes  denote  there  is  a  road  linking  them.  We  want  to  go  from  node  1  to  node  10.  So  node  1  is 
our  source,  which  is  level  0.  We  mark  node  1  as  visited.  We  can  go  to  node  2,  node  3  and  node  4  from  here.  So 
they'll  be  level  (0+1)  =  level  1  nodes.  Now  we'll  mark  them  as  visited  and  work  with  them. 
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The  colored  nodes  are  visited.  The  nodes  that  we're  currently  working  with  will  be  marked  with  pink.  We  won't  visit 
the  same  node  twice.  From  node  2,  node  3  and  node  4,  we  can  go  to  node  6,  node  7  and  node  8.  Let's  mark  them 
as  visited.  The  level  of  these  nodes  will  be  level  (1+1)  =  level  2. 
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If  you  haven't  noticed,  the  level  of  nodes  simply  denote  the  shortest  path  distance  from  the  source.  For  example: 
we've  found  node  8  on  level  2.  So  the  distance  from  source  to  node  8  is  2. 

We  didn't  yet  reach  our  target  node,  that  is  node  10.  So  let's  visit  the  next  nodes,  we  can  directly  go  to  from  node  6, 
node  7  and  node  8. 
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We  can  see  that,  we  found  node  10  at  level  3.  So  the  shortest  path  from  source  to  node  10  is  3.  We  searched  the 
graph  level  by  level  and  found  the  shortest  path.  Now  let's  erase  the  edges  that  we  didn't  use: 
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After  removing  the  edges  that  we  didn't  use,  we  get  a  tree  called  BFS  tree.  This  tree  shows  the  shortest  path  from 
source  to  all  other  nodes. 

So  our  task  will  be,  to  go  from  source  to  level  1  nodes.  Then  from  level  1  to  level  2  nodes  and  so  on  until  we  reach 
our  destination.  We  can  use  queue  to  store  the  nodes  that  we  are  going  to  process.  That  is,  for  each  node  we're 
going  to  work  with,  we'll  push  all  other  nodes  that  can  be  directly  traversed  and  not  yet  traversed  in  the  queue. 

The  simulation  of  our  example: 

First  we  push  the  source  in  the  queue.  Our  queue  will  look  like: 

front 
+ - + 

I  1  I 

+ - + 


The  level  of  node  1  will  be  0.  Ievel[1]  =  0.  Now  we  start  our  BFS.  At  first,  we  pop  a  node  from  our  queue.  We  get 
node  1.  We  can  go  to  node  4,  node  3  and  node  2  from  this  one.  We've  reached  these  nodes  from  node  1.  So 
level[4]  =  level[3]  =  level[2]  =  level[1]  +  1  =  1.  Now  we  mark  them  as  visited  and  push  them  in  the  queue. 

front 

+ - +  + - +  + - + 

I  2  |  |  3  |  |  4  | 

+ - +  + - +  + - + 
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Now  we  pop  node  4  and  work  with  it.  We  can  go  to  node  7  from  node  4.  Ievel[7]  =  level[4]  +  1=2.  We  mark  node  7 
as  visited  and  push  it  in  the  queue. 

front 

+ - +  + - +  + - + 

I  7  |  |  2  |  |  3  | 

+ - +  + - +  + - + 


From  node  3,  we  can  go  to  node  7  and  node  8.  Since  we've  already  marked  node  7  as  visited,  we  mark  node  8  as 
visited,  we  change  level[8]  =  level[3]  +  1=2.  We  push  node  8  in  the  queue. 

front 

+ - +  + - +  + - + 

I  6  |  |  7  |  |  2  | 

+ - +  + - +  + - + 


This  process  will  continue  till  we  reach  our  destination  or  the  queue  becomes  empty.  The  level  array  will  provide  us 
with  the  distance  of  the  shortest  path  from  source.  We  can  initialize  level  array  with  infinity  value,  which  will  mark 
that  the  nodes  are  not  yet  visited.  Our  pseudo-code  will  be: 


Procedure  BFS(Graph,  source): 

Q  =  queue ( ) ; 
level [ ]  =  infinity 
level [source  :=  0 
Q.push(source) 
while  Q  is  not  empty 
u  ->  Q.pop() 

for  all  edges  from  u  to  v  in 
if  level[v]  ==  infinity 
level[v[  :=  level [u. 
Q . push( v) 
end  if 
end  for 
end  while 
Return  level 


Adjacency  list 

+  1 


By  iterating  through  the  level  array,  we  can  find  out  the  distance  of  each  node  from  source.  For  example:  the 
distance  of  node  10  from  source  will  be  stored  in  level[10]. 

Sometimes  we  might  need  to  print  not  only  the  shortest  distance,  but  also  the  path  via  which  we  can  go  to  our 
destined  node  from  the  source.  For  this  we  need  to  keep  a  parent  array.  parent[source]  will  be  NULL.  For  each 
update  in  level  array,  we'll  simply  add  parent  [v]  :=  u  in  our  pseudo  code  inside  the  for  loop.  After  finishing  BFS, 
to  find  the  path,  we'll  traverse  back  the  parent  array  until  we  reach  source  which  will  be  denoted  by  NULL  value. 
The  pseudo-code  will  be: 


Procedure  PrintPath(u) :  //recursive 
if  parent[u  is  not  equal  to  null 
PrintPath(parent[u|) 
end  if 
print  ->  u 


/  Procedure  PrintPath(u) :  //iterative 

|  S  =  Stack() 

|  while  parent[u  is  not  equal  to  null 
|  S.push(u) 

|  u  :=  parent[u 

|  end  while 

|  while  S  is  not  empty 
|  print  ->  S.pop 

end  while 
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Complexity: 


We've  visited  every  node  once  and  every  edges  once.  So  the  complexity  will  be  0(V  +  E)  where  V  is  the  number  of 
nodes  and  E  is  the  number  of  edges. 

Section  41.2:  Finding  Shortest  Path  from  Source  in  a  2D  graph 

Most  of  the  time,  we'll  need  to  find  out  the  shortest  path  from  single  source  to  all  other  nodes  or  a  specific  node  in 
a  2D  graph.  Say  for  example:  we  want  to  find  out  how  many  moves  are  required  for  a  knight  to  reach  a  certain 
square  in  a  chessboard,  or  we  have  an  array  where  some  cells  are  blocked,  we  have  to  find  out  the  shortest  path 
from  one  cell  to  another.  We  can  move  only  horizontally  and  vertically.  Even  diagonal  moves  can  be  possible  too. 
For  these  cases,  we  can  convert  the  squares  or  cells  in  nodes  and  solve  these  problems  easily  using  BFS.  Now  our 
visited,  parent  and  level  will  be  2D  arrays.  For  each  node,  we'll  consider  all  possible  moves.  To  find  the  distance  to 
a  specific  node,  we'll  also  check  whether  we  have  reached  our  destination. 

There  will  be  one  additional  thing  called  direction  array.  This  will  simply  store  the  all  possible  combinations  of 
directions  we  can  go  to.  Let's  say,  for  horizontal  and  vertical  moves,  our  direction  arrays  will  be: 

+ - + - + - + - + - + 

|  dx  |  1  |  -1  |  0  |  0  | 

+ - + - + - + - + - + 

|  dy  |  0  |  0  |  1  |  -1  | 

+ - + - + - + - + - + 


Flere  dx  represents  move  in  x-axis  and  dy  represents  move  in  y-axis.  Again  this  part  is  optional.  You  can  also  write 
all  the  possible  combinations  separately.  But  it's  easier  to  handle  it  using  direction  array.  There  can  be  more  and 
even  different  combinations  for  diagonal  moves  or  knight  moves. 

The  additional  part  we  need  to  keep  in  mind  is: 

•  If  any  of  the  cell  is  blocked,  for  every  possible  moves,  we'll  check  if  the  cell  is  blocked  or  not. 

•  We'll  also  check  if  we  have  gone  out  of  bounds,  that  is  we've  crossed  the  array  boundaries. 

•  The  number  of  rows  and  columns  will  be  given. 

Our  pseudo-code  will  be: 

Procedure  BFS2D(Graph,  blocksign,  row,  column): 
for  i  f  rom  1  to  row 

for  j  from  1  to  column 

visited [ i | [ j ]  :=  false 
end  for 
end  for 

visited[source.x] [source. y]  :=  true 
level[source.x] [source. y]  :=  0 
Q  =  queue() 

Q.push(source) 
m  :=  dx.size 
while  Q  is  not  empty 
top  :=  Q.pop 
for  i  f  rom  1  to  m 

temp.x  :=  top.x  +  dx[i 
temp.y  :=  top.y  +  dy [ i 

if  temp  is  inside  the  row  and  column  and  top  doesn't  equal  to  blocksign 
visited[temp.x] [temp.y]  :=  true 
level [temp . x] [temp.y]  :=  level[top.x] [top.y]  +  1 
Q.push(temp) 
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end  if 
end  for 
end  while 
Return  level 

As  we  have  discussed  earlier,  BFS  only  works  for  unweighted  graphs.  For  weighted  graphs,  we'll  need  Dijkstra's 
algorithm.  For  negative  edge  cycles,  we  need  Bellman-Ford's  algorithm.  Again  this  algorithm  is  single  source 
shortest  path  algorithm.  If  we  need  to  find  out  distance  from  each  nodes  to  all  other  nodes,  we'll  need  Floyd- 
Warshall's  algorithm. 

Section  41.3:  Connected  Components  Of  Undirected  Graph 
Using  BFS 

BFS  can  be  used  to  find  the  connected  components  of  an  undirected  graph.  We  can  also  find  if  the  given  graph  is 
connected  or  not.  Our  subsequent  discussion  assumes  we  are  dealing  with  undirected  graphs.The  definition  of  a 
connected  graph  is: 


A  graph  is  connected  if  there  is  a  path  between  every  pair  of  vertices. 


Following  is  a  connected  graph. 


Following  graph  is  not  connected  and  has  2  connected  components: 

1 .  Connected  Component  1 :  {a,b,c,d,e} 

2.  Connected  Component  2:  {f} 
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BFS  is  a  graph  traversal  algorithm.  So  starting  from  a  random  source  node,  if  on  termination  of  algorithm,  all  nodes 
are  visited,  then  the  graph  is  connected, otherwise  it  is  not  connected. 

Pseudocode  for  the  algorithm. 

boolean  isConnected (Graph  g) 

{ 

BFS(v)//v  is  a  random  source  node. 
if (allVisited(g) ) 

{ 

return  true 

} 

else  return  false 

} 

C  implementation  for  finding  the  whether  an  undirected  graph  is  connected  or  not: 

#include<stdio . h> 

#include<stdlib . h> 

#def ine  MAXVERTICES  100 

void  enqueue( int) ; 
int  deque( )  ; 

int  isConnected (char  **graph,int  noOfVertices); 
void  BFS(char  **graph,int  vertex, int  noOfVertices); 
int  count  =  0; 

//Queue  node  depicts  a  single  Queue  element 
//It  is  NOT  a  graph  node, 
struct  node 
{ 

int  v 

struct  node  *next; 

}; 


typedef  struct  node  Node 
typedef  struct  node  *Nodeptr; 

Nodeptr  Qfront  =  NULL; 

Nodeptr  Qrear  =  NULL; 

char  *visited ; //array  that  keeps  track  of  visited  vertices, 
int  main() 
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{ 


int  n,e;//n  is  number  of  vertices,  e  is  number  of  edges, 
int  i,  j ; 

char  **graph  //adjacency  matrix 


printf( "Enter  number  of  vertices:"); 
scanf ( "%d" , &n)  ; 

if (n  <0  ||  n  >  MAXVERTICES) 

{ 

fprintf (stderr,  "Please  enter  a  valid  positive  integer  from  1  to  %d" , MAXVERTICES) ; 
return  -1  ; 

} 

graph  =  malloc(n  *  sizeof(char  *)); 
visited  =  malloc(n*sizeof (char) ) ; 

for(i  =  0i  <  n;++i) 

{ 

graph[i]  =  malloc(n*sizeof (int) ) ; 

visited[i]  =  ' N //initially  all  vertices  are  not  visited. 
for(j  =  0;j  <  n  ++ j ) 
graph [i] [j ]  =  0 ; 

} 

printf( "enter  number  of  edges  and  then  enter  them  in  pairs:"); 
scanf ( "%d" , &e) ; 

for(i  =  0;i  <  e;++i) 

{ 

int  u,v 

scanf ( "%d%d" , &u ,  &v) ; 
graph [u-1 ] [ v-1 ]  =  1  ; 
graph[v-1 ] [u-1  ]  =  1  ; 

} 

if (isConnected( graph, n) ) 

printf("The  graph  is  connected"); 
else  printf("The  graph  is  NOT  connected\n" )  ; 


void  enqueue(int  vertex) 

{ 

if(Qfront  ==  NULL) 

{ 

Qfront  =  malloc(sizeof (Node) )  ; 

Qfront->v  =  vertex; 

Qfront->next  =  NULL; 

Qrear  =  Qfront; 

} 

else 

{ 

Nodeptr  newNode  =  malloc(sizeof (Node) ) ; 
newNode->v  =  vertex; 
newNode->next  =  NULL; 

Qrear->next  =  newNode 
Qrear  =  newNode 

} 


int  deque() 

{ 
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if(Qfront  ==  NULL) 

{ 

printf("Q  is  empty  ,  returning  -1\n"); 
return  -1  ; 

} 

else 

{ 

int  v  =  Qf ront->v 
Nodeptr  temp=  Qfront; 
if (Qf ront  ==  Qrear) 

{ 

Qfront  =  Qf ront->next ; 

Qrear  =  NULL; 

} 

else 

Qfront  =  Qf ront->next ; 

free (temp) ; 
return  v; 

} 

} 

int  isConnected (char  **graph,int  noOfVertices) 

{ 

int  i 

//let  random  source  vertex  be  vertex  0; 

BFS(graph, 0, noOfVertices) ; 

for(i  =  0  i  <  noOfVertices;++i) 
if (visited [ i]  ==  ' N ' ) 
return  0;//0  implies  false; 

return  1 ;  / / 1  implies  true; 

} 

void  BFS(char  **graph,int  v,int  noOfVertices) 

{ 

int  i, vertex 
visited [v ]  =  ' Y ' ; 
enqueue(v) ; 

while( (vertex  =  deque())  !=  -1) 

{ 

for(i  =  0 , i  <  noOfVertices ;++i) 

if (graph[vertex] [i]  ==  1  &&  visited[i]  ==  ' N ' ) 

{ 

enqueue(i) ; 
visited [ i ]  =  '  Y '  ; 

} 

} 

} 

For  Finding  all  the  Connected  components  of  an  undirected  graph,  we  only  need  to  add  2  lines  of  code  to  the  BFS 
function.  The  idea  is  to  call  BFS  function  until  all  vertices  are  visited. 

The  lines  to  be  added  are: 

printf ( " XnConnected  component  %d\n" , ++count)  ; 

//count  is  a  global  variable  initialized  to  0 
//add  this  as  first  line  to  BFS  function 
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AND 


printf("%d  ",vertex+1); 

add  this  as  first  line  of  while  loop  in  BFS 

and  we  define  the  following  function: 

void  listConnectedComponents(char  **graph,int  noOfVertices) 

{ 

int  i 

for(i  =  0  i  <  noOfVertices ;++i) 

{ 

if (visited [ i]  ==  '  N ' ) 

BFS ( g raph, i, noOfVertices) ; 

} 

} 
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Chapter  42:  Depth  First  Search 

Section  42.1:  Introduction  To  Depth-First  Search 

Depth-first  search  is  an  algorithm  for  traversing  or  searching  tree  or  graph  data  structures.  One  starts  at  the  root 
and  explores  as  far  as  possible  along  each  branch  before  backtracking.  A  version  of  depth-first  search  was 
investigated  in  the  19th  century  French  mathematician  Charles  Pierre  Tremaux  as  a  strategy  for  solving  mazes. 

Depth-first  search  is  a  systematic  way  to  find  all  the  vertices  reachable  from  a  source  vertex.  Like  breadth-first 
search,  DFS  traverse  a  connected  component  of  a  given  graph  and  defines  a  spanning  tree.  The  basic  idea  of  depth- 
first  search  is  methodically  exploring  every  edge.  We  start  over  from  a  different  vertices  as  necessary.  As  soon  as 
we  discover  a  vertex,  DFS  starts  exploring  from  it  (unlike  BFS,  which  puts  a  vertex  on  a  queue  so  that  it  explores 
from  it  later). 

Let's  look  at  an  example.  We'll  traverse  this  graph: 


We'll  traverse  the  graph  following  these  rules: 

•  We'll  start  from  the  source. 

•  No  node  will  be  visited  twice. 

•  The  nodes  we  didn't  visit  yet,  will  be  colored  white. 

•  The  node  we  visited,  but  didn't  visit  all  of  its  child  nodes,  will  be  colored  grey. 

•  Completely  traversed  nodes  will  be  colored  black. 

Let's  look  at  it  step  by  step: 
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step-  5 


step- 6 
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We  can  see  one  important  keyword.  That  is  backedge.  You  can  see.  5-1  is  called  backedge.  This  is  because,  we're 
not  yet  done  with  node-1,  so  going  from  another  node  to  node-1  means  there's  a  cycle  in  the  graph.  In  DFS,  if  we 
can  go  from  one  gray  node  to  another,  we  can  be  certain  that  the  graph  has  a  cycle.  This  is  one  of  the  ways  of 
detecting  cycle  in  a  graph.  Depending  on  source  node  and  the  order  of  the  nodes  we  visit,  we  can  find  out  any  edge 
in  a  cycle  as  backedge.  For  example:  if  we  went  to  5  from  1  first,  we'd  have  found  out  2-1  as  backedge. 

The  edge  that  we  take  to  go  from  gray  node  to  white  node  are  called  tree  edge.  If  we  only  keep  the  tree  edge's  and 
remove  others,  we'll  get  DFS  tree. 

In  undirected  graph,  if  we  can  visit  a  already  visited  node,  that  must  be  a  backedge.  But  for  directed  graphs,  we 
must  check  the  colors.  If  and  only  if  we  can  go  from  one  gray  node  to  another  gray  node,  that  is  called  a  backedge. 

In  DFS,  we  can  also  keep  timestamps  for  each  node,  which  can  be  used  in  many  ways  (e.g.:  Topological  Sort). 

1 .  When  a  node  v  is  changed  from  white  to  gray  the  time  is  recorded  in  d[v]. 
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2.  When  a  node  v  is  changed  from  gray  to  black  the  time  is  recorded  in  f[v]. 

Here  d[]  means  discovery  time  and  f[]  means  finishing  time.  Our  pesudo-code  will  look  like: 

Procedure  DFS(G)  : 
for  each  node  u  in  V [ G ] 
color[u  :=  white 
parent[u]  :=  NULL 
end  for 
time  :=  0 

for  each  node  u  in  V [ G ] 
if  color[u  ==  white 
DFS  Visit(u) 
end  if 

end  for 

Procedure  DFS  Visit(u): 
color[u  :=  gray 
time  :=  time  +  1 
d[u]  :=  time 

for  each  node  v  adjacent  to  u 
if  color[vi  ==  white 
parent[v]  :=  u 
DFS  Visit(v) 
end  if 
end  for 

color[u  :=  black 
time  :=  time  +  1 
f[u]  :=  time 

Complexity: 

Each  nodes  and  edges  are  visited  once.  So  the  complexity  of  DFS  is  0(V+E),  where  V  denotes  the  number  of  nodes 
and  E  denotes  the  number  of  edges. 

Applications  of  Depth  First  Search: 

•  Finding  all  pair  shortest  path  in  an  undirected  graph. 

•  Detecting  cycle  in  a  graph. 

•  Path  finding. 

•  Topological  Sort. 

•  Testing  if  a  graph  is  bipartite. 

•  Finding  Strongly  Connected  Component. 

•  Solving  puzzles  with  one  solution. 
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Chapter  43:  Hash  Functions 

Section  43.1:  Hash  codes  for  common  types  in  C# 

The  hash  codes  produced  by  GetHashCode( )  method  for  built-in  and  common  C#  types  from  the  System 
namespace  are  shown  below. 

Boolean 


1  if  value  is  true,  0  otherwise. 

Byte.  Ulnt16.  Int32.  Ulnt32.  Single 
Value  (if  necessary  casted  to  Int32). 

SBvte 

( (int)m_value  A  (int)m_value  <<  8); 

Char 

(int)m_value  A  ( (int)m_value  <<  16); 

Int16 

( (int) ( (ushort)m_value)  A  ( ( ( int)m_value)  <<  16)); 

Int64.  Double 

Xor  between  lower  and  upper  32  bits  of  64  bit  number 

(unchecked( (int) ( (long)m_value) )  A  (int) (m_value  >>  32)); 

Ulnt64.  DateTime.  TimeSpan 
( (int)m_value)  A  (int) (m_value  >>  32); 

Decimal 

((((int  *)&dbl) [0] )  &  0xFFFFFFF0)  A  ((int  *)&dbl)[1]; 

Object 

RuntimeFlelpers .  GetFlashCode(this) ; 

The  default  implementation  is  used  sync  block  index. 

String 

Hash  code  computation  depends  on  the  platform  type  (Win32  or  Win64),  feature  of  using  randomized  string 
hashing,  Debug  /  Release  mode.  In  case  ofWin64  platform: 

int  hashl  =  5381  ; 
int  hash2  =  hashl ; 

int  c ; 

char  *s  =  src; 

while  ((c  =  s [0] )  !=  0)  { 

hashl  =  ((hashl  <<  5)  +  hashl)  A  c; 
c  =  s  [  1  ]  ; 
if  (c  ==  0) 
break 

hash2  =  ((hash2  <<  5)  +  hash2)  A  c; 
s  +=  2; 

} 
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return  hashl  +  (hash2  *  1566083941); 

ValueTvpe 


The  first  non-static  field  is  look  for  and  get  it's  hashcode.  If  the  type  has  no  non-static  fields,  the  hashcode  of  the 
type  returns.  The  hashcode  of  a  static  member  can't  be  taken  because  if  that  member  is  of  the  same  type  as  the 
original  type,  the  calculating  ends  up  in  an  infinite  loop. 

Nullable<T> 

return  hasValue  ?  value .GetHashCode( )  :  0; 

Array 

int  ret  =  0; 

for  (int  i  =  (Length  >=  8  ?  Length  -  8  :  0);  i  <  Length  i++) 

{ 

ret  =  ((ret  <<  5)  +  ret)  A  comparer .GetHashCode(GetValue(i) ) ; 

} 

References 

•  GitHub  .Net  Core  CLR 

Section  43.2:  Introduction  to  hash  functions 

Hash  function  h( )  is  an  arbitrary  function  which  mapped  data  x  e  X  of  arbitrary  size  to  value  y  e  Y  of  fixed  size:  y 
=  h(x).  Good  hash  functions  have  follows  restrictions: 

•  hash  functions  behave  likes  uniform  distribution 

•  hash  functions  is  deterministic,  h  (x)  should  always  return  the  same  value  for  a  given  x 

•  fast  calculating  (has  runtime  0(1 )) 

In  general  case  size  of  hash  function  less  then  size  of  input  data:  |  y  |  <  |  x  | .  Hash  functions  are  not  reversible  or  in 
other  words  it  may  be  collision:  a  xl,  x2  e  x,  xl  *  x2  h(xl)  =  h(x2).  X  may  be  finite  or  infinite  set  and  Y  is 
finite  set. 

Hash  functions  are  used  in  a  lot  of  parts  of  computer  science,  for  example  in  software  engineering,  cryptography, 
databases,  networks,  machine  learning  and  so  on.  There  are  many  different  types  of  hash  functions,  with  differing 
domain  specific  properties. 

Often  hash  is  an  integer  value.  There  are  special  methods  in  programmning  languages  for  hash  calculating.  For 
example,  in  C#  GetHashCode( )  method  for  all  types  returns  Int32  value  (32  bit  integer  number).  In  Java  every  class 
provides  hashCode( )  method  which  return  int.  Each  data  type  has  own  or  user  defined  implementations. 

Hash  methods 

There  are  several  approaches  for  determinig  hash  function.  Without  loss  of  generality,  lets  x  eX  =  {zeZ:  z  > 

0}  are  positive  integer  numbers.  Often  m  is  prime  (not  too  close  to  an  exact  power  of  2). 

Method  Hash  function 

Division  method  h(x)  =  x  mod  m 

Multiplication  method  h(x)  =  Lm  (xA  mod  1 )  j ,  A  e  {z  e  R:  0  <  z  <  1 } 

Hash  table 

Hash  functions  used  in  hash  tables  for  computing  index  into  an  array  of  slots.  Hash  table  is  data  structure  for 
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implementing  dictionaries  (key-value  structure).  Good  implemented  hash  tables  have  0(1 )  time  for  the  next 
operations:  insert,  search  and  delete  data  by  key.  More  than  one  keys  may  hash  to  the  same  slot.  There  are  two 
ways  for  resolving  collision: 

1 .  Chaining:  linked  list  is  used  for  storing  elements  with  the  same  hash  value  in  slot 

2.  Open  addressing:  zero  or  one  element  is  stored  in  each  slot 


The  next  methods  are  used  to  compute  the  probe  sequences  required  for  open  addressing 

Method  Formula 

Linear  probing  h(x,  i)  =  (h'(x)  +  i)  mod  m 

Quadratic  probing h(x,  i)  =  (h'(x)  +  c1*i  +  c2*iA2)  mod  m 

Double  hashing  h(x,  i)  =  (hl(x)  +  i*h2(x))  mod  m 

Where  i  e  {0,  1,  ...,  m-1},h'(x),  hl(x),  h2(x)  are  auxiliary  hash  functions,  cl ,  c2  are  positive  auxiliary 
constants. 

Examples 

Lets  x  e  U{1,  1000},  h  =  x  mod  m.  The  next  table  shows  the  hash  values  in  case  of  not  prime  and  prime.  Bolded 
text  indicates  the  same  hash  values. 

x  m  =  100  (not  prime)  m  =  101  (prime) 


723  23 

16 

103  3 

2 

738  38 

31 

292  92 

90 

61  61 

61 

87  87 

87 

995  95 

86 

549  49 

44 

991  91 

82 

757  57 

50 

920  20 

11 

626  26 

20 

557  57 

52 

831  31 

23 

619  19 

13 

Links 

•  Thomas  H.  Cormen,  Charles  E.  Leiserson,  Ronald  L.  Rivest,  Clifford  Stein.  Introduction  to  Algorithms. 

•  Overview  of  Hash  Tables 


•  Wolfram  MathWorld  -  Hash  Function 
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Chapter  44:  Travelling  Salesman 

Section  44.1:  Brute  Force  Algorithm 

A  path  through  every  vertex  exactly  once  is  the  same  as  ordering  the  vertex  in  some  way.  Thus,  to  calculate  the 
minimum  cost  of  travelling  through  every  vertex  exactly  once,  we  can  brute  force  every  single  one  of  the  N! 
permutations  of  the  numbers  from  1  to  N. 

Psuedocode 

minimum  =  INF 

for  all  permutations  P 

current  =  0 

for  i  f  rom  0  to  N-2 

current  =  current  +  cost[P[i] ] [P[i+1 ] ]  <-  Add  the  cost  of  going  from  1  vertex  to  the  next 

current  =  current  +  cost [  P[ N-1  ] ]  [  P[ 0] ]  <-  Add  the  cost  of  going  from  last  vertex  to  the 

first 

if  current  <  minimum  <-  Update  minimum  if  necessary 

minimum  =  current 

output  minimum 

Time  Complexity 

There  are  N !  permutations  to  go  through  and  the  cost  of  each  path  is  calculated  in  0(N),  thus  this  algorithm  takes 
0(N  *  N! )  time  to  output  the  exact  answer. 

Section  44.2:  Dynamic  Programming  Algorithm 

Notice  that  if  we  consider  the  path  (in  order): 

(1,2, 3, 4, 6, 0,5, 7) 
and  the  path 
(1,2, 3, 5, 0,6, 7, 4) 

The  cost  of  going  from  vertex  1  to  vertex  2  to  vertex  3  remains  the  same,  so  why  must  it  be  recalculated?  This  result 
can  be  saved  for  later  use. 

Let  dp  [bitmask]  [vertex]  represent  the  minimum  cost  of  travelling  through  all  the  vertices  whose  corresponding 
bit  in  bitmask  is  set  to  1  ending  at  vertex.  For  example: 

dp[ 12] [2] 

12  =  1100 
A  A 

vertices  3210 

Since  12  represents  1 100  in  binary,  dp  [12]  [2]  represents  going  through  vertices  2  and  3  in  the  graph  with  the  path 
ending  at  vertex  2. 
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Thus  we  can  have  the  following  algorithm  (C++  implementation): 


int  cost[N] [N] ; 

int  memo[1  <<  N ] [ N ] ; 

int  TSP(int  bitmask,  int  pos){ 


//Adjust  the  value  of  N  if  needed 
//Set  everything  here  to  -7 


int  cost  =  INF 


if  (bitmask  ==  ((1  <<  N)  -  1)){ 

return  cost[pos] [0]  ; 


//All  vertices  have  been  explored 
//Cost  to  go  back 


} 


if  (memo[bitmask] [pos]  !=  -1){ 
return  memo [bitmask  [pos! 


//If  this  has  already  been  computed 

//Just  return  the  value,  no  need  to  recompute 


} 


for  (int  i  =  0;  i  <  N;  ++i){ 

if  ((bitmask  &  (1  <<  i))  ==  0){ 


//For  every  vertex 

//If  the  vertex  has  not  been  visited 
|  (1  <<  i)  ,  i)  +  cost[pos][i  );  //Visit  the  vertex 


cost  =  min(cost,TSP(bitmask 


} 

} 

memo [bitmask  |  [pos]  =  cost; 
return  cost  ; 


//Save  the  result 


} 

//Call  TSP(l,e) 

This  line  may  be  a  little  confusing,  so  lets  go  through  it  slowly: 
cost  =  min(cost,TSP(bitmask  |  (1  <<  i)  ,  i)  +  cost[pos] [i] ) ; 

Here,  bitmask  |  (1  <<  i)  sets  the  ith  bit  of  bitmask  to  1 ,  which  represents  that  the  ith  vertex  has  been  visited.  The 
i  after  the  comma  represents  the  new  pos  in  that  function  call,  which  represents  the  new  "last"  vertex, 
cost  [  pos  ]  [  i  ]  is  to  add  the  cost  of  travelling  from  vertex  pos  to  vertex  i. 

Thus,  this  line  is  to  update  the  value  of  cost  to  the  minimum  possible  value  of  travelling  to  every  other  vertex  that 
has  not  been  visited  yet. 

Time  Complexity 

The  function  TSP(bitmask,  pos)  has  2AN  values  for  bitmask  and  N  values  for  pos.  Each  function  takes  0(N)  time  to 
run  (the  for  loop).  Thus  this  implementation  takes  0(NA2  *  2AN)  time  to  output  the  exact  answer. 
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Chapter  45:  Knapsack  Problem 

Section  45.1:  Knapsack  Problem  Basics 

The  Problem:  Given  a  set  of  items  where  each  item  contains  a  weight  and  value,  determine  the  number  of  each  to 
include  in  a  collection  so  that  the  total  weight  is  less  than  or  equal  to  a  given  limit  and  the  total  value  is  as  large  as 
possible. 

Pseudo  code  for  Knapsack  Problem 

Given: 

1.  Values(array  v) 

2.  Weights(array  w) 

3.  Number  of  distinct  items(n) 

4.  Capacity(W) 

for  j  from  0  to  W  do 
m[0,  j]  :=  0 
for  i  from  1  to  n  do 

for  j  from  0  to  W  do 
if  w[i]  >  j  then 

m[i,  j ]  : =  m[ i-1 ,  j  ] 

else 

m[i,  j]  :=  max(m[i-1,  j],  m[i-1,  j-w[i]]  +  v[i) 

A  simple  implementation  of  the  above  pseudo  code  using  Python: 

def  knapSack(W,  wt,  val,  n)  : 

K  =  [[0  for  x  in  range(W+1)]  for  x  in  range(n+1)] 
for  i  in  range(n+1 )  : 

for  w  in  range(W+1 ) : 
if  i==0  or  w==0 : 

K [ i ] [w]  =  0 
elif  wt [ i  1 ]  <=  w 

K [ i ] [ w]  =  max(val[i  1]  +  K[ i -1 ] [w  wt [ i -1 ] ] ,  K[ i-1 ] [ w] ) 

else 

K [ i ] [w]  =  K[i-1 ] [w] 
return  K[n] [W] 

val  =  [60,  100,  120] 
wt  =  [10,  20,  30] 

W  =  50 

n  =  len(val) 

print (knapSack(W,  wt,  val,  n)) 

Running  the  code:  Save  this  in  a  file  named  knapSack. py 

$  python  knapSack. py 

220 

Time  Complexity  of  the  above  code:  0(nW)  where  n  is  the  number  of  items  and  W  is  the  capacity  of  knapsack. 

Section  45.2:  Solution  Implemented  in  C# 

public  class  KnapsackProblem 

{ 
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private  static  int  Knapsack(int  w,  int[]  weight,  int[]  value,  int  n) 

{ 

int  i 

int[,]  k  =  new  int[n  +1,  w  +  1]; 
for  (i  =  0;  i  <=  n  i++) 

{ 

int  b 

for  (b  =  0;  b  <=  w  b++) 

{ 

if  (i==0  [ |  b==0) 

{ 

k  [  i ,  b  ]  =  0  ; 

} 

else  if  (weight[i  -  1 ]  <=  b) 

{ 

k[i,  b]  =  Math . Max(value[ i  -  1]  +  k[i  -  1,  b  weight[i  -  1]],  k[i  -  1,  b]); 

} 

else 

{ 

k[i,  b]  =  k [ i  -  1,  b] 

} 

} 

} 

return  k[n,  w] ; 


public  static  int  Main(int  nltems,  int[]  weights,  int[]  values) 

{ 

int  n  =  values . Length ; 

return  Knapsack(nltems,  weights,  values,  n)  ; 

} 
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Chapter  46:  Matrix  Exponentiation 

Section  46.1:  Matrix  Exponentiation  to  Solve  Example 
Problems 

Find  f(n):  nth  Fibonacci  number.  The  problem  is  quite  easy  when  n  is  relatively  small.  We  can  use  simple  recursion, 
f  (n)  =  f  (n-l )  +  f(n -2),  or  we  can  use  dynamic  programming  approach  to  avoid  the  calculation  of  same  function 
over  and  over  again.  But  what  will  you  do  if  the  problem  says,  Given  0  <  n  <  109,  find  f(n)  mod  999983?  Dynamic 
programming  will  fail,  so  how  do  we  tackle  this  problem? 

First  let's  see  how  matrix  exponentiation  can  help  to  represent  recursive  relation. 

Prerequisites: 

•  Given  two  matrices,  know  how  to  find  their  product.  Further,  given  the  product  matrix  of  two  matrices,  and 
one  of  them,  know  how  to  find  the  other  matrix. 

•  Given  a  matrix  of  size  d  X  d,  know  how  to  find  its  nth  power  in  0(d3log(n)). 

Patterns: 

At  first  we  need  a  recursive  relation  and  we  want  to  find  a  matrix  M  which  can  lead  us  to  the  desired  state  from  a 
set  of  already  known  states.  Let's  assume  that,  we  know  the  k  states  of  a  given  recurrence  relation  and  we  want  to 
find  the  (k+1)th  state.  Let  M  be  a  k  X  k  matrix,  and  we  build  a  matrix  A:[k  X  1]  from  the  known  states  of  the 
recurrence  relation,  now  we  want  to  get  a  matrix  B:[k  X  1]  which  will  represent  the  set  of  next  states,  i.  e.  M  X  A  =  B 
as  shown  below: 


1  f(n)  I 

1  f (n+1 )  | 

1  f(n  1)  | 

1  f(n)  I 

1  f(n  2)  | 

|  .  | 

=  1  f(n  1)  | 

1  .  1 

1  f(n  k)  | 

| f (n  k+1 ) | 

So,  if  we  can  design  M  accordingly,  our  job  will  be  done!  The  matrix  will  then  be  used  to  represent  the  recurrence 
relation. 

Type  1: 

Let's  start  with  the  simplest  one,  f(n)  =  f(n-1)  +  f(n-2) 

We  get,  f(n+1)  =  f(n)  +  f(n  1). 

Let's  assume,  we  know  f  (n)  and  f  (n  1 );  We  want  to  find  out  f  (n+1 ). 

From  the  situation  stated  above,  matrix  A  and  matrix  B  can  be  formed  as  shown  below: 

Matrix  A  Matrix  B 

I  f(n)  |  |  f (n+1 )  | 

I  f (n-1 )  |  |  f (n)  | 

[Note:  Matrix  Awill  be  always  designed  in  such  a  waythat,  every  state  on  which  f  (n+1 )  depends,  will  be  present] 
Now,  we  need  to  design  a  2X2  matrix  M  such  that,  it  satisfies  M  X  A  =  B  as  stated  above. 

The  first  element  of  B  is  f  (n+1 )  which  is  actually  f(n)  +  f  (n  -1 ).  To  get  this,  from  matrix  A,  we  need,  1  X  f(n)  and  1 
X  f(n-1).  So  the  first  row  of  M  will  be  [1  1], 

|  1  1  |  X  |  f(n)  |  =  |  f (n+1 )  | 

|  -  |  |  f (n-1 )  |  |  -  | 
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[Note:  —  means  we  are  not  concerned  about  this  value.] 

Similarly,  2nd  item  of  B  is  f  (n)  which  can  be  got  by  simply  taking  1  X  f(n)  from  A,  so  the  2nd  row  of  M  is  [1  0], 

|  -  |  X  |  f(n)  |  =  |  -  | 

|  1  0  |  |  f (n-1 )  |  |  f(n)  | 

Then  we  get  our  desired  2X2  matrix  M. 

|  1  1  |  X  |  f(n)  |  =  |  f (n+1 )  | 

|  1  0  |  |  f (n-1 )  |  |  f(n)  | 

These  matrices  are  simply  derived  using  matrix  multiplication. 

Type  2: 

Let's  make  it  a  little  complex:  find  f(n)  =  a  X  f(n-1)  +  b  X  f(n -2),  where  a  and  b  are  constants. 

This  tells  us,  f  (n+1 )  =  a  X  f(n)  +  b  X  f(n-1). 

By  this  far,  this  should  be  clear  that  the  dimension  of  the  matrices  will  be  equal  to  the  number  of  dependencies,  i.e. 
in  this  particular  example,  again  2.  So  for  A  and  B,  we  can  build  two  matrices  of  size  2X1: 

Matrix  A  Matrix  B 

I  f(n)  |  |  f (n+1 )  | 

I  f (n-1 )  |  |  f (n)  | 

Nowforf(n+1)  =  a  X  f(n)  +  b  X  f  (n-1 ),  we  need  [a,  b]  in  the  first  row  of  objective  matrix  M.  And  for  the  2nd 
item  in  B,  i.e.  f  (n)  we  already  have  that  in  matrix  A,  so  we  just  take  that,  which  leads,  the  2nd  row  of  the  matrix  M 
to  [1  0],  This  time  we  get: 

|  a  b  |  X  |  f (n)  |  =  |  f (n+1 )  | 

|  1  0  |  |  f (n-1 )  |  |  f (n)  | 

Pretty  simple,  eh? 

Type  3: 

If  you've  survived  through  to  this  stage,  you've  grown  much  older,  now  let's  face  a  bit  complex  relation:  find  f  (n)  = 
a  X  f ( n- 1 )  +  c  X  f(n-3)? 

Ooops!  A  few  minutes  ago,  all  we  saw  were  contiguous  states,  but  here,  the  state  f(n-2)  is  missing.  Now? 

Actually  this  is  not  a  problem  anymore,  we  can  convert  the  relation  as  follows:  f(n)  =  a  X  f(n-l)  +  0  X  f(n-2)  + 
c  X  f(n-3),  deducing  f  (n+1 )  =  a  X  f(n)  +  0  X  f  ( n - 1 )  +  c  X  f(n-2).  Now,  we  see  that,  this  is  actually  a  form 
described  in  Type  2.  So  here  the  objective  matrix  M  will  be  3  X  3,  and  the  elements  are: 

|  a  0  c  |  |  f (n)  |  |  f (n+1 )  | 

|  1  0  0  |  X  |  f (n-1 )  |  =  |  f (n)  | 

|  0  1  0  |  |  f (n-2)  |  |  f (n-1 )  | 

These  are  calculated  in  the  same  way  as  type  2,  if  you  find  it  difficult,  try  it  on  pen  and  paper. 

Type  4: 

Life  is  getting  complex  as  hell,  and  Mr,  Problem  now  asks  you  to  find  f(n)  =  f(n-l)  +  f(n-2)  +  c  where  c  is  any 
constant. 

Now  this  is  a  new  one  and  all  we  have  seen  in  past,  after  the  multiplication,  each  state  in  A  transforms  to  its  next 
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state  in  B. 


f(n)  =  f(n-1)  +  f(n  2)  +  c 
f(n+1)  =  f(n)  +  f(n  1)  +  c 
f(n+2)  =  f(n+1)  +  f(n)  +  c 
.  so  on 


So  ,  normally  we  can't  get  it  through  previous  fashion,  but  how  about  we  add  c  as  a  state: 

I  f(n)  |  |  f (n+1 )  | 

M  X  |  f(n  1)  |  =  |  f(n)  | 

I  c  |  |  c  | 

Now,  its  not  much  hard  to  design  M.  Here's  how  its  done,  but  don't  forget  to  verify: 

Mill  |  f (n)  |  |  f (n+1 )  | 

|  1  0  0  |  X  |  f(n  1)  |  =  |  f (n)  | 

|  0  0  1  |  |  c  |  |  c  | 

Type  5: 

Let's  put  it  altogether:  find  f(n)  =  a  X  f(n-1)  +  c  X  f(n-3)  +  d  X  f(n-4)  +  e.  Let's  leave  it  as  an  exercise  for 
you.  First  try  to  find  out  the  states  and  matrix  M.  And  check  if  it  matches  with  your  solution.  Also  find  matrix  A  and 

B. 


|  a  0  c  d  1  | 

|1  0  0  0  0| 

|0  1  0  0  0| 

|0  0  1  0  0| 

10  0  0  0  1  | 

Type  6: 

Sometimes  the  recurrence  is  given  like  this: 

f(n)  =  f(n-1)  ->  if  n  is  odd 

f(n)  =  f(n-2)  ->  if  n  is  even 

In  short: 

f(n)  =  (n&l)  X  f(n  1)  +  ( ! ( n&l ) )  X  f(n  2) 

Here,  we  can  split  the  functions  in  the  basis  of  odd  even  and  keep  2  different  matrix  for  both  of  them  and  calculate 
them  separately. 

Type  7: 

Feeling  little  too  confident?  Good  for  you.  Sometimes  we  may  need  to  maintain  more  than  one  recurrence,  where 
they  are  interested.  For  example,  let  a  recurrence  re;atopm  be: 

9 (n)  =  2g(n  1)  +  2g(n  2)  +  f(n) 

Here,  recurrence  g(n)  is  dependent  upon  f  (n)  and  this  can  be  calculated  in  the  same  matrix  but  of  increased 
dimensions.  From  these  let's  at  first  design  the  matrices  A  and  B. 
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Matrix  A 
I  g(n)  I 
I  g(n  1)  I 
I  f(n+1)  | 
I  f(n)  I 


Matrix  B 

I  9 (n+1 )  | 
I  g(n)  I 
I  f(n+2)  | 
I  f (n+1 )  | 


Here,  g(n+1)  =  2g(n-1)  +  f(n+l)  and  f(n+2)  =  2f(n+l)  +  2f(n).  Now,  using  the  processes  stated  above,  we 
can  find  the  objective  matrix  M  to  be: 

|  2  2  1  0  | 

|  1  0  0  0  | 

|  0  0  2  2  | 

|  0  0  1  0  | 


So,  these  are  the  basic  categories  of  recurrence  relations  which  are  used  to  solveby  this  simple  technique. 
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Chapter  47:  Equation  Solving 

Section  47.1:  Linear  Equation 

There  are  two  classes  of  methods  for  solving  Linear  Equations: 

1 .  Direct  Methods:  Common  characteristics  of  direct  methods  are  that  they  transform  the  original  equation 
into  equivalent  equations  that  can  be  solved  more  easily,  means  we  get  solve  directly  from  an  equation. 

2.  Iterative  Method:  Iterative  or  Indirect  Methods,  start  with  a  guess  of  the  solution  and  then  repeatedly  refine 
the  solution  until  a  certain  convergence  criterion  is  reached.  Iterative  methods  are  generally  less  efficient 
than  direct  methods  because  large  number  of  operations  required.  Example- Jacobi's  Iteration  Method, 
Gauss-Seidal  Iteration  Method. 


Implementation  in  C- 

/ /Implementation  of  Jacobi's  Method 

void  JacobisMethod(int  n,  double  x [ n ] ,  double  b [ n J ,  double  a[n  [n  ){ 

double  Nx[n];  //modified  form  of  variables 
int  rootFound=0;  //flag 

int  i,  j ; 

while( ! rootFound)  { 

for  (i=0;  i<n  i++){  //calculation 

Nx[i]=b[i]  ; 

for(j=0;  j<n  ]++){ 

if(i!=j)  Nx [ i ]  =  Nx[i]-a[i] [j]*x[ j]  ; 

} 

Nx [ i ]  =  Nx [ i ]  /  a [ i ] [i]  ; 

} 

rootFound  =  1  ;  //verification 

for(i=0;  i<n  i++){ 

if ( ! (  ( Nx [ i ] - x [ i ] )/x[i]  >  -0.000001  &&  ( Nx [ i ] -x [ i ] ) /x [ i ]  <  0.000001  )){ 

rootFound=0 ; 

break 

} 

} 

for(i=0;  i<n  i++){  / /evaluation 

x [ i ] =  Nx [ i  ]  ; 

} 

} 

return 

} 

//Implementation  of  Gauss-Seidal  Method 

void  GaussSeidalMethod(int  n,  double  x[n',  double  b[n],  double  a[n][n]){ 
double  Nx[n];  //modified  form  of  variables 
int  rootFound=0;  //flag 

int  i,  j ; 

for  ( i = 0 ;  i<n  i++){  //initialization 

Nx [ i ] =x [ i ] ; 

} 
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while( ! rootFound) { 

for(i=0;  i<n  i++){  //calculation 

Nx[i]-b[i] ; 

for(j=0;  j<n;  ]++){ 

if(i!=j)  Nx[i]  =  Nx[i]  a [ i ] [ j ]*Nx[ j ] ; 

} 

Nx[i]  =  Nx[i]  /  a [ i ] [ i ] ; 

} 

rootFound  =  1  ;  //verification 

for(i=0;  i<n  i++){ 

if ( ! (  (Nx[i]-x[i] )/x[i]  >  -0.000001  &&  (Nx[ i] -x[i] ) /x[i] 
rootFound=0 ; 

break 

} 

} 

for(i=0;  i<n  i++){  //evaluation 

x[i]=Nx[i]  ; 

} 

} 

return 

} 

//Print  array  with  comma  separation 

void  print(int  n,  double  x[n]){ 
int  i 

for(i=0;  i<n  i++){ 

printf ( "%lf ,  ",  x [ i ] )  ; 

} 

printf ( " \n\n" )  ; 

return 

} 

int  main(){ 

//equation  initialization 

int  n=3;  //number  of  variables 

double  x [ n ] ;  //variables 

double  b[n],  //constants 

a[n] [n]  ;  //arguments 

//assign  values 

a [0 ] [ 0] =8 ;  a [ 0 ] [1 ] =2 ;  a [ 0 ] [ 2 ] =-2 ;  b [ 0 ] =  8 ;  //8xi+2x2-2x3+8=0 

a [  1  ]  [ 0]  =1  ;  a [  1  ]  [1  ]  =  - 8 ;  a [  1  ]  [ 2 ]  =3 ;  b[1]  =  -4;  //xi-8x2+3x3-4=0 

a [ 2 ] [ 0 ] =2 ;  a [ 2 ] [1 ] = 1 ;  a [2] [2] =9 ;  b [ 2 ] = 1 2 ;  //2xi+x2+9x3+72=0 


int  i 

for  ( i = 0 ;  i<n  i++){  //initialization 

x [ i ] =0 ; 

} 

JacobisMethod (n ,  x,  b,  a); 
print(n,  x) ; 


for(i=0;  i<n  i++){ 


//initialization 


<  0.000001  )){ 
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x [ i ] =0 ; 

} 

GaussSeidalMethod (n ,  x,  b,  a); 
print (n ,  x) ; 

return  0; 

} 

Section  47.2:  Non-Linear  Equation 

An  equation  of  the  type  f  (x)=0  is  either  algebraic  or  transcendental.  These  types  of  equations  can  be  solved  by 
using  two  types  of  methods- 

1 .  Direct  Method:  This  method  gives  the  exact  value  of  all  the  roots  directly  in  a  finite  number  of  steps. 

2.  Indirect  or  Iterative  Method:  Iterative  methods  are  best  suited  for  computer  programs  to  solve  an 
equation.  It  is  based  on  the  concept  of  successive  approximation.  In  Iterative  Method  there  are  two  ways  to 
solve  an  equation- 

°  Bracketing  Method:  We  take  two  initial  points  where  the  root  lies  in  between  them.  Example- 
Bisection  Method,  False  Position  Method. 

°  Open  End  Method:  We  take  one  or  two  initial  values  where  the  root  may  be  any-where.  Example- 
Newton-Raphson  Method,  Successive  Approximation  Method,  Secant  Method. 


Implementation  in  C: 

III  Here  define  different  functions  to  work  with 
#define  f(x)  (  ( (x)*(x)*(x) )  -  (x)  -  2  ) 

#define  f2(x)  (  (3*(x)*(x))  -  1  ) 

#define  g(x)  (  cbrt(  (x)  +  2  )  ) 


/** 

*  Takes  two  initial  values  and  shortens  the  distance  by  both  side. 
**/ 

double  BisectionMethod( ) { 
double  root=0; 

double  a=1 ,  b=2 ; 
double  c=0; 

int  loopCounter=0 ; 
if (f (a)*f (b)  <  0){ 
while(1 )  { 

loopCounter++ ; 
c=(a+b) /2 ; 

if (f(c)<0. 00001  &&  f(c)>-0.00001){ 
root=c ; 
break ; 

} 

if ( (f (a) )*(f (c) )  <  0){ 
b=c ; 

}else{ 
a=c  ; 

} 
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} 

} 

printf("It  took  %d  loops. \n",  loopCounter)  ; 
return  root; 

} 

/** 

*  Takes  two  initial  values  and  shortens  the  distance  by  single  side. 

**/ 

double  FalsePosition( ) { 
double  root=0; 

double  a=1 ,  b=2 ; 
double  c=0; 

int  loopCounter=0 ; 
if (f (a)*f (b)  <  0){ 
while(1 )  { 

loopCounter++ ; 

c=(a*f(b)  -  b*f(a))  /  (f(b)  -  f(a)); 

/*/printf( "%lf\t  %lf  \n",  c,  f (c)) ;/**/// / test 
if (f(c)<0. 00001  &&  f(c)>-0.00001){ 
root=c ; 
break ; 

} 

if ( (f (a) )*(f (c) )  <  0){ 
b=c ; 

}else{ 
a=c ; 

} 

} 

} 

printf("It  took  %d  loops. \n",  loopCounter); 
return  root; 

} 

/** 

*  Uses  one  initial  value  and  gradually  takes  that  value  near  to  the  real  one. 
**/ 

double  NewtonRaphson( ) { 
double  root=0; 

double  xl =1  ; 
double  x2=0; 

int  loopCounter=0 ; 
while(1 ) { 

loopCounter++ ; 

x2  =  xl  -  (f (xl )/f2(x1 ) ) ; 

/*/printf( "%lf  \t  Xlf  \n" ,  x2,  f(x2) );/**/// / test 

if (f(x2)<0. 00001  &&  f (x2) >-0 . 00001  )  { 
root=x2 
break ; 

} 
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x1=x2 

} 

printf("It  took  %d  loops. \n",  loopCounter)  ; 
return  root; 

} 

/** 

*  Uses  one  initial  value  and  gradually  takes  that  value  near  to  the  real  one. 
**/ 

double  FixedPoint ( ) { 
double  root=0; 
double  x=1 ; 

int  loopCounter=0 ; 
while(1 ) { 

loopCounter++ ; 

if (  (x-g(x) )  <0.00001  &&  (x-g (x) )  >-0.00001){ 
root  =  x; 
break ; 

} 

/*/printf("Xlf  \t  %lf  \n",  g(x),  x- (g(x))) ;/**/// / test 
x=g(x) ; 

} 

printf("It  took  %d  loops. \n",  loopCounter); 
return  root; 

} 

/** 

*  uses  two  initial  values  &  both  value  approaches  to  the  root. 

**/ 

double  Secant(){ 
double  root=0; 

double  x0=1 ; 
double  x1=2; 
double  x2=0; 

int  loopCounter=0 ; 
while(1 ) { 

loopCounter++ ; 

/*/printf( "%lf  \t  %lf  \t  %lf  \n",  x0,  x7,  f (xl ));/**/// / test 

if (f(x1 )<0. 00001  &&  f(x1)>-0.00001){ 
root=x1  ; 
break  ; 

} 

x2  =  ( (x0*f (xl ) )- (x1*f (x0) ) )  /  (f (xl ) -f (x0) ) ; 

x0=x1 ; 
xl =x2 ; 

} 

printf("It  took  %d  loops. \n",  loopCounter); 
return  root  ; 

} 
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int  main(){ 

double  root; 

root  =  BisectionMethod( ) ; 

printf( "Using  Bisection  Method  the  root  is:  %lf  \n\n",  root); 
root  =  FalsePosition( )  ; 

printf( "Using  False  Position  Method  the  root  is:  %lf  \n\n",  root); 
root  =  NewtonRaphson( )  ; 

printf( "Using  Newton-Raphson  Method  the  root  is:  %lf  \n\n",  root); 
root  =  FixedPoint ( )  ; 

printf( "Using  Fixed  Point  Method  the  root  is:  %lf  \n\n",  root); 
root  =  Secant(); 

printf( "Using  Secant  Method  the  root  is:  %lf  \n\n",  root)  ; 
return  0; 
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Chapter  48:  Longest  Common 
Subsequence 

Section  48.1:  Longest  Common  Subsequence  Explanation 

One  of  the  most  important  implementations  of  Dynamic  Programming  is  finding  out  the  Longest  Common 
Subsequence.  Let's  define  some  of  the  basic  terminologies  first. 

Subsequence: 

A  subsequence  is  a  sequence  that  can  be  derived  from  another  sequence  by  deleting  some  elements  without 
changing  the  order  of  the  remaining  elements.  Let's  say  we  have  a  string  ABC.  If  we  erase  zero  or  one  or  more  than 
one  character  from  this  string  we  get  the  subsequence  of  this  string.  So  the  subsequences  of  string  ABC  will  be 
{"A",  "B",  "C",  "AB",  "AC",  "BC",  "ABC",  "  "}.  Even  if  we  remove  all  the  characters,  the  empty  string  will  also  be  a 
subsequence.  To  find  out  the  subsequence,  for  each  characters  in  a  string,  we  have  two  options  -  either  we  take  the 
character,  or  we  don't.  So  if  the  length  of  the  string  is  n,  there  are  2n  subsequences  of  that  string. 

Longest  Common  Subsequence: 

As  the  name  suggest,  of  all  the  common  subsequencesbetween  two  strings,  the  longest  common  subsequence(LCS) 
is  the  one  with  the  maximum  length.  For  example:  The  common  subsequences  between  "HELLOM"  and  "HMLD" 
are  "H",  "HL",  "HM"  etc.  Here  "HLL"  is  the  longest  common  subsequence  which  has  length  3. 

Brute-Force  Method: 

We  can  generate  all  the  subsequences  of  two  strings  using  backtracking.  Then  we  can  compare  them  to  find  out  the 
common  subsequences.  After  we'll  need  to  find  out  the  one  with  the  maximum  length.  We  have  already  seen  that, 
there  are  2n  subsequences  of  a  string  of  length  n.  It  would  take  years  to  solve  the  problem  if  our  n  crosses  20-25. 

Dynamic  Programming  Method: 

Let's  approach  our  method  with  an  example.  Assume  that,  we  have  two  strings  abcdaf  and  acbcf.  Let's  denote 
these  with  si  and  s2.  So  the  longest  common  subsequence  of  these  two  strings  will  be  "abcf",  which  has  length  4. 
Again  I  remind  you,  subsequences  need  not  be  continuous  in  the  string.  To  construct  "abcf",  we  ignored  "da"  in  si 
and  "c"  in  s2.  How  do  we  find  this  out  using  Dynamic  Programming? 

We'll  start  with  a  table  (a  2D  array)  having  all  the  characters  of  si  in  a  row  and  all  the  characters  of  s2  in  column. 
Here  the  table  is  0-indexed  and  we  put  the  characters  from  1  to  onwards.  We'll  traverse  the  table  from  left  to  right 
for  each  row.  Our  table  will  look  like: 


+ - +- 

I  clr  | 

+ - +- 


0 


+ - 


1 


+ - 


2 


+  — 


3 


b 


+ - 


4 


+ - 


0 

_  _  + - 

I  a 
-  -  +  — 


-  -  +  — 


_  _  +  — 


_  _  +  — 


-  -  +  — 


-  -  +  — 


1 

.  _  +  — 

I  b 
-  -  +  — 


-  -  +  — 


-  -  +  — 


-  -  +  — 


-  -  +  — 


-  -  +  — 


3 

_  +  — 

I  d 
_  +  — 


_  +  — 


-+ — 


_+  — 


_+  — 


_+  — 


4 

-  -  +  — 

I  a 
-  -  +  — 


-  -  +  — 


-  -  +  — 


-  -  +  — 


-  -  +  — 


-  -  +  — 
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5  I  f  I  I  I  I  I  I  I 
+ - + - + - + - + - + - + - + - + 


Here  each  row  and  column  represent  the  length  of  the  longest  common  subsequence  between  two  strings  if  we 
take  the  characters  of  that  row  and  column  and  add  to  the  prefix  before  it.  For  example:  Table[2][3]  represents  the 
length  of  the  longest  common  subsequence  between  "ac"  and  "abc". 

The  0-th  column  represents  the  empty  subsequence  of  si.  Similarly  the  0-th  row  represents  the  empty 
subsequence  of  s2.  If  we  take  an  empty  subsequence  of  a  string  and  try  to  match  it  with  another  string,  no  matter 
how  long  the  length  of  the  second  substring  is,  the  common  subsequence  will  have  0  length.  So  we  can  fill-up  the  0- 
th  rows  and  0-th  columns  with  0's.  We  get: 


0 

1 

2 

3 

4 

5 

+  - 

-  +  - 

-  +  - 

-  + — 

-  -  + - 

-  +  - 

-  -  + - 

_  +  — 

-  -  + 

1 

chr 

1 

i 

a 

1  b 

1  c 

i 

d 

1  a 

1  f 

i 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

_  +  — 

--  + - 

-  +  - 

--  + - 

_  +  — 

-  -  + 

0 

1 

i 

0 

1 

0  | 

0  | 

0 

1 

0  | 

0  | 

0 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

_  +  — 

--  + - 

-  +  - 

--  + - 

_  +  — 

-  -  + 

1 

1 

a 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  — 

-  -  + - 

-  +  - 

-  -  + - 

_  +  — 

-  -  + 

2 

1 

c 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  — 

-  -  + - 

-  +  - 

--  + - 

_  +  — 

-  -  + 

3 

1 

b 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

_  +  — 

--  + - 

-  +  - 

--  + - 

_  +  — 

-  -  + 

4 

1 

c 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

_  +  — 

--  + - 

-  +  - 

--  + - 

_  +  — 

-  -  + 

5 

1 

f 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  + — 

-  -  + - 

-  +  - 

-  -  + - 

_  +  — 

-  -  + 

Let's  begin.  When  we're  filling  Table[1][1],  we're  asking  ourselves,  if  we  had  a  string  a  and  another  string  a  and 
nothing  else,  what  will  be  the  longest  common  subsequence  here?  The  length  of  the  LCS  here  will  be  1 .  Now  let's 
look  at  Table[1][2],  We  have  string  ab  and  string  a.  The  length  of  the  LCS  will  be  1 .  As  you  can  see,  the  rest  of  the 
values  will  be  also  1  for  the  first  row  as  it  considers  only  string  a  with  abed,  abeda,  abedaf.  So  our  table  will  look 
like: 


0 

1 

2 

3 

4 

5 

+  - 

-  +  - 

-  +  - 

-  + - 

-  -  + - 

-  +  - 

-  -  + - 

_  +  — 

-  -  + 

i 

chr 

1 

i 

a 

1  b 

1  c 

i 

d 

1  a 

1  f 

i 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

_  +  — 

--  + - 

-  +  - 

--  + - 

_  +  — 

-  -  + 

0 

1 

i 

0 

1 

0  | 

0  | 

0 

1 

0  | 

0  | 

0 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  + — 

-  -  + - 

-  +  - 

-  -  + - 

_  +  — 

-  -  + 

1 

1 

a 

i 

0 

1 

1  | 

1  | 

1 

1 

1  | 

1  | 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  + — 

-  -  + - 

-  +  - 

--  + - 

_  +  — 

-  -  + 

2 

1 

c 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-+- 

— 

-  +  - 

— 

-  + — 

-  -  + - 

-  +  - 

--  + - 

_  +  — 

-  -  + 

3 

1 

b 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

_  +  — 

--  + - 

-  +  - 

--  + - 

_  +  — 

-  -  + 

4 

1 

c 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-+- 

— 

-  +  - 

— 

_  +  — 

--  + - 

-  +  - 

--  + - 

_  +  — 

-  -  + 

5 

1 

f 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  + — 

-  -  + - 

-  +  - 

-  -  + - 

_  +  — 

-  -  + 

For  row  2,  which  will  now  include  c.  For  Table[2][1]  we  have  ac  on  one  side  and  a  on  the  other  side.  So  the  length  of 
the  LCS  is  1 .  Where  did  we  get  this  1  from?  From  the  top,  which  denotes  the  LCS  a  between  two  substrings.  So  what 
we  are  saying  is,  if  si  [2]  and  s2[1]  are  not  same,  then  the  length  of  the  LCS  will  be  the  maximum  of  the  length  of 
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LCS  at  the  top,  or  at  the  left.  Taking  the  length  of  the  LCS  at  the  top  denotes  that,  we  don't  take  the  current 
character  from  s2.  Similarly,  Taking  the  length  of  the  LCS  at  the  left  denotes  that,  we  don't  take  the  current 
character  from  si  to  create  the  LCS.  We  get: 


0 

1 

2 

3 

4 

5 

+  - 

-  +  - 

-  +  - 

-  + — 

-  -  + - 

-  +  - 

-  -  + - 

_  +  — 

-  -  + 

1 

chr 

1 

i 

a 

1  b 

1  c 

i 

d 

1  a 

1  f 

i 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

_  +  — 

--  + - 

-  +  - 

--  + - 

_  +  — 

-  -  + 

0 

1 

i 

0 

1 

0  | 

0  | 

0 

1 

0  | 

0  | 

0 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  — 

-  -  + - 

-  +  - 

-  -  + - 

_  +  — 

-  -  + 

1 

1 

a 

i 

0 

1 

1  | 

1  | 

1 

1 

1  | 

1  | 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  + — 

-  -  + - 

-  +  - 

-  -  + - 

_  +  — 

-  -  + 

2 

1 

c 

i 

0 

1 

1  | 

1 

1 

1 

1 

+  - 

— 

-+- 

— 

-  +  - 

— 

_  +  — 

--  + - 

-  +  - 

--  + - 

_  +  — 

-  -  + 

3 

1 

b 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

_  +  — 

--  + - 

-  +  - 

--  + - 

_  +  — 

-  -  + 

4 

1 

c 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  + — 

-  -  + - 

-  +  - 

-  -  + - 

_  +  — 

-  -  + 

5 

1 

f 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  + — 

--  + - 

-  +  - 

-  -  + - 

_  +  — 

-  -  + 

So  our  first  formula  will  be: 


if  s2[i  is  not  equal  to  s1[j] 

Table [ i ] [ j ]  =  max(Table[i  -1 ] [ j ] ,  Table[i] [ j-1  ] 
endif 


Moving  on,  for  Table[2][2]  we  have  string  ab  and  ac.  Since  c  and  b  are  not  same,  we  put  the  maximum  of  the  top  or 
left  here.  In  this  case,  it's  again  1 .  After  that,  for  Table[2][3]  we  have  string  abc  and  ac.  This  time  current  values  of 
both  row  and  column  are  same.  Now  the  length  of  the  LCS  will  be  equal  to  the  maximum  length  of  LCS  so  far  +  1 . 
How  do  we  get  the  maximum  length  of  LCS  so  far?  We  check  the  diagonal  value,  which  represents  the  best  match 
between  ab  and  a.  From  this  state,  for  the  current  values,  we  added  one  more  character  to  si  and  s2  which 
happened  to  be  the  same.  So  the  length  of  LCS  will  of  course  increase.  We'll  put  1  +  1  =  2  in  Table[2][3],  We  get, 


0 

1 

2 

3 

4 

5 

+  - 

-  +  - 

-  +  - 

_  + - 

-  +  - 

-  +  - 

-  -  + 

i 

chr 

1 

i 

a 

1  b 

i 

c 

1 

d 

1 

a 

1 

f 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

_  +  — 

— 

-  +  - 

— 

-  +  - 

-  -  + 

0 

1 

i 

0 

1 

0  | 

0 

1 

0 

1 

0 

1 

0 

1 

0 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  — 

— 

-  +  - 

— 

-  +  - 

-  -  + 

1 

1 

a 

i 

0 

1 

1  | 

1 

1 

1 

1 

1 

1 

1 

1 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  + — 

— 

-  +  - 

— 

-  +  - 

-  -  + 

2 

1 

c 

i 

0 

1 

1  | 

1 

1 

2 

1 

1 

1 

+  - 

— 

-+- 

— 

-  +  - 

— 

_  +  — 

— 

-  +  - 

— 

-  +  - 

-  -  + 

3 

1 

b 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

_  +  — 

— 

-  +  - 

— 

-  +  - 

-  -  + 

4 

1 

c 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  — 

— 

-  +  - 

— 

-  +  - 

-  -  + 

5 

1 

f 

i 

0 

1 

1 

1 

1 

1 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  + — 

— 

-  +  - 

-  -  +  - 

— 

-  +  - 

-  -  + 

So  our  second  formula  will  be: 


if  s2[i  equals  to  s1[j] 

T  able [ i ] [ j ]  =  T able [i-1 ] [j-1 ]  +  1 
endif 
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We  have  defined  both  the  cases.  Using  these  two  formulas,  we  can  populate  the  whole  table.  After  filling  up  the 
table,  it  will  look  like  this: 


0 

1 

2 

3 

4 

5 

+  - 

-  +  - 

-  +  - 

-  +  - 

--  + - 

-  +  - 

-  +  - 

-  -  + 

1 

chr 

1 

i 

a 

1 

b 

1  c 

i 

d 

1 

a 

1 

f 

1 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

-  -  + - 

-  +  - 

— 

-  +  - 

-  -  + 

0 

1 

i 

0 

1 

0 

1 

0  | 

0 

1 

0 

1 

0 

1 

0 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

-  -  + - 

-  +  - 

— 

-  +  - 

-  -  + 

1 

1 

a 

i 

0 

1 

1 

1 

1  | 

1 

1 

1 

1 

1 

1 

1 

+  - 

— 

-+- 

— 

-  +  - 

— 

-  +  - 

--  + - 

-  +  - 

— 

-  +  - 

-  -  + 

2 

1 

c 

i 

0 

1 

1 

1 

1  | 

2 

1 

2 

1 

2 

1 

2 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

--  + - 

-  +  - 

— 

-  +  - 

-  -  + 

3 

1 

b 

i 

0 

1 

1 

1 

2  1 

2 

1 

2 

1 

2 

1 

2 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

--  + - 

-  +  - 

— 

-  +  - 

-  -  + 

4 

1 

c 

i 

0 

1 

1 

1 

2  1 

3 

1 

3 

1 

3 

1 

3 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

--  + - 

-  +  - 

— 

-  +  - 

-  -  + 

5 

1 

f 

i 

0 

1 

1 

1 

2  1 

3 

1 

3 

1 

3 

1 

4 

+  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

--  + - 

-  +  - 

— 

-  +  - 

-  -  + 

The  length  of  the  LCS  between  si  and  s2  will  be  Table[5][6]  =  4.  Here,  5  and  6  are  the  length  of  s2  and  si 
respectively.  Our  pseudo-code  will  be: 

Procedure  LCSlength (si ,  s2) : 

Table [ 0 ] [ 0 ]  =  0 
for  i  f  rom  1  to  si . length 
Table[0] [i]  =  0 
endfor 

for  i  f  rom  1  to  s2. length 
Table [ i ] [ 0]  =  0 
endfor 

for  i  f  rom  1  to  s2. length 

for  j  f  rom  1  to  si . length 
if  s2[i  equals  to  s1[j] 

Table[i][j]  =  Table[i-1 ] [ j-1 ]  +  1 

else 

Table [ i ] [ j ]  =  max(Table [ i -1  ]  [ j  ] ,  Tablet i] [ j -1  ] ) 
endif 
endfor 
endfor 

Return  Tablet s2. length] [si .length] 

The  time  complexity  for  this  algorithm  is:  O(mn)  where  m  and  n  denotes  the  length  of  each  strings. 

How  do  we  find  out  the  longest  common  subsequence?  We'll  start  from  the  bottom-right  corner.  We  will  check 
from  where  the  value  is  coming.  If  the  value  is  coming  from  the  diagonal,  that  is  if  Table[i-1]|J-1]  is  equal  to 
Table[i]|J]  - 1,  we  push  either  s2[i]  or  si  [j]  (both  are  the  same)  and  move  diagonally.  If  the  value  is  coming  from  top, 
that  means,  if  Table[i-1]|j]  is  equal  to  Table[i][j],  we  move  to  the  top.  If  the  value  is  coming  from  left,  that  means,  if 
Table[i]|j-1]  is  equal  to  Table[i][j],  we  move  to  the  left.  When  we  reach  the  leftmost  or  topmost  column,  our  search 
ends.  Then  we  pop  the  values  from  the  stack  and  print  them.  The  pseudo-code: 

Procedure  PrintLCS(LCSlength,  si,  s2) 
temp  :=  LCSlength 
S  =  stack() 
i  :=  s2. length 
j  : =  si . length 

while  i  is  not  equal  to  0  and  j  is  not  equal  to  0 

if  Table[i  1][j-1]  ==  Table [ i ] [ j ]  -  1  and  si  [j ]==s2[i] 
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S.push(s1 [ j  ] ) 

//or  S.push(s2[i]) 

i  :=  i  -  1 

j  :=  j  -  1 

else  if  Table [ i  -1  ]  [ 

j ]  ==  Table[ i] [ j  ] 

i  :=  i  1 


else 

3  :=  j-1 

endif 

endwhile 

while  S  is  not  empty 
print (S . pop) 
endwhile 

Point  to  be  noted:  if  both  Table[i-1]|j]  and  Table[i][j-1]  is  equal  to  Table[i]|J]  and  Table[i-1]|j-1]  is  not  equal  to 
Table[i][j]  - 1,  there  can  be  two  LCS  for  that  moment.  This  pseudo-code  doesn't  consider  this  situation.  You'll  have 
to  solve  this  recursively  to  find  multiple  LCSs. 

The  time  complexity  for  this  algorithm  is:  0(max(m,  n)). 
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Chapter  49:  Longest  Increasing 
Subsequence 

Section  49.1:  Longest  Increasing  Subsequence  Basic 
Information 

The  Longest  Increasing  Subsequence  problem  is  to  find  subsequence  from  the  give  input  sequence  in  which 
subsequence's  elements  are  sorted  in  lowest  to  highest  order.  All  subsequence  are  not  contiguous  or  unique. 

Application  of  Longest  Increasing  Subsequence: 

Algorithms  like  Longest  Increasing  Subsequence,  Longest  Common  Subsequence  are  used  in  version  control 
systems  like  Git  and  etc. 

Simple  form  of  Algorithm: 

1 .  Find  unique  lines  which  are  common  to  both  documents. 

2.  Take  all  such  lines  from  the  first  document  and  order  them  according  to  their  appearance  in  the  second 
document. 

3.  Compute  the  LIS  of  the  resulting  sequence  (by  doing  a  Patience  Sort),  getting  the  longest  matching  sequence 
of  lines,  a  correspondence  between  the  lines  of  two  documents. 

4.  Recurse  the  algorithm  on  each  range  of  lines  between  already  matched  ones. 

Now  let  us  consider  a  simpler  example  of  the  LCS  problem.  Here,  input  is  only  one  sequence  of  distinct  integers 
al ,  a2,  . . . ,  an .,  and  we  want  to  find  the  longest  increasing  subsequence  in  it.  For  example,  if  input  is  7,3,8,4,2,6 
then  the  longest  increasing  subsequence  is  3,4,6. 

The  easiest  approach  is  to  sort  input  elements  in  increasing  order,  and  apply  the  LCS  algorithm  to  the  original  and 
sorted  sequences.  However,  if  you  look  at  the  resulting  array  you  would  notice  that  many  values  are  the  same,  and 
the  array  looks  very  repetitive.  This  suggest  that  the  LIS  (longest  increasing  subsequence)  problem  can  be  done 
with  dynamic  programming  algorithm  using  only  one-dimensional  array. 

Pseudo  Code: 

1 .  Describe  an  array  of  values  we  want  to  compute. 

For  1  <=  i  <=  n,  let  A(i)  be  the  length  of  a  longest  increasing  sequence  of  input.  Note  that  the  length  we  are 
ultimately  interested  in  is  max{A(i)  1 1  <  i  <  n}. 

2.  Give  a  recurrence. 

Fori  <=  i  <=  n,  A(i)  =  1  +  max{A(j)|1  <  j  <  iandinput(j)  <  input  ( i) } . 

3.  Compute  the  values  of  A. 

4.  Find  the  optimal  solution. 

The  following  program  uses  A  to  compute  an  optimal  solution.  The  first  part  computes  a  value  m  such  that  A(m)  is 
the  length  of  an  optimal  increasing  subsequence  of  input.  The  second  part  computes  an  optimal  increasing 
subsequence,  but  for  convenience  we  print  it  out  in  reverse  order.  This  program  runs  in  time  O(n),  so  the  entire 
algorithm  runs  in  time  0(nA2). 

Part  1: 

m  <-  1 

for  i  2. .n 

if  A(i)  >  A(m)  then 
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m  <-  1 


end  if 

end  for 

Part  2: 

put  a 

while  A(m)  >  1  do 
i  <-  m-1 

while  not(ai  <  am  and  A(i)  =  A(m)-1)  do 
i  <-  i-1 

end  while 
m  <-  i 
put  a 
end  while 

Recursive  Solution: 

Approach  1: 

LIS(A[1 . . n] ) : 

if  (n  =  0)  then  return  0 
m  =  LIS ( A [ 1 ..(n  -  1)]) 

B  is  subsequence  of  A[1..(n  -  1)]  with  only  elements  less  than  a[n 
(*  let  h  be  size  of  B,  h  <  n-1  *) 
m  =  max(m,  1  +  LIS(B[1 . .h] ) ) 

Output  m 

Time  complexity  in  Approach  1  :  0(n*2An) 

Approach  2: 

LIS(A[1 . . n ] ,  x) : 

if  (n  =  0)  then  return  0 
m  =  LIS(A[1 . . (n  -  1 ) ] ,  x) 
if  (A[n]  <  x)  then 

m  =  max(m,  1  +  LIS(A[1..(n  -  1)],  A[n])) 

Output  m 

MAIN( A[ 1 . .n]) : 

return  LIS(A[1..n],  °°) 

Time  Complexity  in  Approach  2:  0(nA2) 

Approach  3: 

LIS(A[1 . . n] ) : 

if  (n  =  0)  return  0 

m  =  1 

for  i  =  1  to  n  -  1  do 

if  ( A [ i ]  <  A[n] )  then 

m  =  max(m,  1  +  LIS(A[ 1 . . i] ) ) 

return  m 

MAIN( A[ 1 . .n]) : 

return  LIS(A['I  .  .i]) 
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Time  Complexity  in  Approach  3:  0(nA2) 

Iterative  Algorithm: 

Computes  the  values  iteratively  in  bottom  up  fashion. 

LIS(A[1 . . n] ) : 

Array  L [ 1 . . n ] 

(*  L[i]  =  value  of  LIS  ending(A[1  . . i] )  *) 
for  i  =  1  to  n  do 
L[i]  =  1 

for  j  =  1  to  i  -  1  do 
if  (A [ j ]  <  A[i] )  do 

L  [  i  ]  =  max(L[ i] ,  1  +  L[j]) 

return  L 

MAIN( A[ 1 . . n ] )  : 

L  =  LIS(A[1 . .n] ) 

return  the  maximum  value  in  L 

Time  complexity  in  Iterative  approach:  0(nA2) 

Auxiliary  Space:  0(n) 

Lets  take  (0,  8,  4, 12,  2, 10,  6, 14, 1,  9,  5, 13,  3, 11,  7, 15}  as  input.  So,  Longest  Increasing  Subsequence  for  the  given 
input  is  (0,  2,  6,  9, 11, 15}. 


GoalKicker.com  -  Algorithms  Notes  for  Professionals 


228 


Chapter  50:  Dynamic  Time  Warping 

Section  50.1:  Introduction  To  Dynamic  Time  Warping 

Dynamic  Time  Warping  (DTW)  is  an  algorithm  for  measuring  similarity  between  two  temporal  sequences  which  may 
vary  in  speed.  For  instance,  similarities  in  walking  could  be  detected  using  DTW,  even  if  one  person  was  walking 
faster  than  the  other,  or  if  there  were  accelerations  and  decelerations  during  the  course  of  an  observation.  It  can  be 
used  to  match  a  sample  voice  command  with  others  command,  even  if  the  person  talks  faster  or  slower  than  the 
prerecorded  sample  voice.  DTW  can  be  applied  to  temporal  sequences  of  video,  audio  and  graphics  data-indeed, 
any  data  which  can  be  turned  into  a  linear  sequence  can  be  analyzed  with  DTW. 

In  general,  DTW  is  a  method  that  calculates  an  optimal  match  between  two  given  sequences  with  certain 
restrictions.  But  let's  stick  to  the  simpler  points  here.  Let's  say,  we  have  two  voice  sequences  Sample  and  Test,  and 
we  want  to  check  if  these  two  sequences  match  or  not.  Here  voice  sequence  refers  to  the  converted  digital  signal  of 
your  voice.  It  might  be  the  amplitude  or  frequency  of  your  voice  that  denotes  the  words  you  say.  Let's  assume: 

Sample  =  {1,  2,  3,  5,  5,  5,  6} 

Test  =  {1 ,  1 ,  2,  2,  3,  5} 

We  want  to  find  out  the  optimal  match  between  these  two  sequences. 

At  first,  we  define  the  distance  between  two  points,  d(x,  y)  where  x  and  y  represent  the  two  points.  Let, 
d(x,  y )  =  | x  y |  //absolute  difference 

Let's  create  a  2D  matrix  Table  using  these  two  sequences.  We'll  calculate  the  distances  between  each  point  of 
Sample  with  every  points  of  Test  and  find  the  optimal  match  between  them. 


I  0 
+ - 

I  1 
+ - 

I  2 
+ - 

I  3 
+ - 

I  5 
+ - 

I  5 
+ - 

I  5 
+ - 

I  6 


Here,  Table[i]|J]  represents  the  optimal  distance  between  two  sequences  if  we  consider  the  sequence  up  to 
Sample[i]  and  Test[j],  considering  all  the  optimal  distances  we  observed  before. 

For  the  first  row,  if  we  take  no  values  from  Sample,  the  distance  between  this  and  Test  will  be  infinity.  So  we  put 
infinity  on  the  first  row.  Same  goes  for  the  first  column.  If  we  take  no  values  from  Test,  the  distance  between  this 
one  and  Sample  will  also  be  infinity.  And  the  distance  between  0  and  0  will  simply  be  0.  We  get, 
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+ . + . + . -  + . -  + . -  + . -  + . -  + . -  + 


I  0  I  1  I  1  I  2  |  2  |  3  |  5  | 


+  -  - 


+  -  - 


+  -  - 


+  -  - 


+  -  - 


+  -  - 


+  -  - 


+  -  - 


+  -  - 


- + . +  -  - 

0  |  0  | 
- + . -  +  -  - 

1  |  inf  | 

- + - +  -  - 

2  |  inf  | 

- + . +  -  - 

3  |  inf  | 

- + . +  -  - 

5  |  inf  | 

- + - +  _  - 

5  |  inf  | 

- + - +  -  - 

5  |  inf  | 

- + . +  -  - 

6  |  inf  | 

—  + . +  -  - 


- +  -  _ 

inf  | 
- +  -  _ 


- +  _  _ 


- +  _  _ 


- +  _  _ 


- +  _ . 


- +  -  - 


- +  -  _ 

inf  | 
- +  -  _ 


- +  _ . 


- +  _  _ 


- +  _  _ 


- +  -  - 


- +  -  - 


- + . -  + . -  + . -  + 

inf  |  inf  |  inf  |  inf  | 
- + - + - + - + 

i  i  i  i 

- + - + - + - + 

i  i  i  i 

- + . -  + . -  + . -  + 

i  i  i  i 

- + . -  + . -  + - + 

i  i  i  i 

- + - + - + - + 

i  i  i  i 

- + - + - + - + 

i  i  i  i 

- + . -  + . -  + . -  + 

i  i  i  i 

— + - + - + - + 


Now  for  each  step,  we'll  consider  the  distance  between  each  points  in  concern  and  add  it  with  the  minimum 
distance  we  found  so  far.  This  will  give  us  the  optimal  distance  of  two  sequences  up  to  that  position.  Our  formula 
will  be, 

Table[i][j]  :=  d(i,  j)  +  min(Table[ i-1  ] [ j ] ,  Table[i  1  ]  [  j-1  ] ,  Table[ i] [ j-1 ] ) 

For  the  first  one,  d(1, 1)  =  0,  Table[0][0]  represents  the  minimum.  So  the  value  of  Table[1][1]  will  be  0  +  0  =  0.  For 
the  second  one,  d(1,  2)  =  0.  Table[1][1]  represents  the  minimum.  The  value  will  be:  Table[1][2]  =  0  +  0  =  0.  If  we 
continue  this  way,  after  finishing,  the  table  will  look  like: 


+  -  - 

— 

-+- 

— 

-  +  - 

— 

-+- 

— 

-  +  - 

— 

-+- 

— 

-+- 

— 

-+- 

— 

-  + 

1 

1 

0 

1 

1 

1 

1 

1 

2 

1 

2 

1 

3 

1 

5 

1 

+  -  - 

— 

-  +  - 

— 

-  +  - 

— 

-+- 

— 

-  +  - 

— 

-+- 

— 

-+- 

— 

-  +  - 

— 

-  + 

1 

0 

1 

0 

1 

inf 

1 

inf 

1 

inf 

1 

inf 

1 

inf 

1 

inf 

1 

+  -  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  + 

1 

1 

1 

inf 

1 

0 

1 

0 

1 

1 

1 

2 

1 

4 

1 

8 

1 

+  -  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  + 

1 

2 

1 

inf 

1 

1 

1 

1 

1 

0 

1 

0 

1 

1 

1 

4 

1 

+  -  - 

— 

-+- 

— 

-  +  - 

— 

-+- 

— 

-  +  - 

— 

-+- 

— 

-  +  - 

— 

-+- 

— 

-  + 

1 

3 

1 

inf 

1 

3 

1 

3 

1 

1 

1 

1 

1 

0 

1 

2 

1 

+  -  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-+- 

— 

-  +  - 

— 

-  + 

1 

5 

1 

inf 

1 

7 

1 

7 

1 

4 

1 

4 

1 

2 

1 

0 

1 

+  -  - 

— 

-+- 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-+- 

— 

-  +  - 

— 

-+- 

— 

-  + 

1 

5 

1 

inf 

1 

11 

1 

11 

1 

7 

1 

7 

1 

4 

1 

0 

1 

+  -  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  + 

1 

5 

1 

inf 

1 

15 

1 

15 

1 

10 

1 

10 

1 

6 

1 

0 

1 

+  -  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  +  - 

— 

-  + 

1 

6 

1 

inf 

1 

20 

1 

20 

1 

14 

1 

14 

1 

9 

1 

1 

1 

+  -  - 

— 

-+- 

— 

-  +  - 

— 

-+- 

— 

-  +  - 

— 

-  +  - 

— 

-+- 

— 

-+- 

— 

-  + 

The  value  at  Table[7][6]  represents  the  maximum  distance  between  these  two  given  sequences.  Flere  1  represents 
the  maximum  distance  between  Sample  and  Test  is  1. 

Now  if  we  backtrack  from  the  last  point,  all  the  way  back  towards  the  starting  (0,  0)  point,  we  get  a  long  line  that 
moves  horizontally,  vertically  and  diagonally.  Our  backtracking  procedure  will  be: 

if  Table[i-1  ] [ j-1  ]  <=  Table [ i-1 ][ j ]  and  Table[i-1 ] [ j-1 ]  <=  Table[i] [ j-1 ] 
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i  :=  i  -  1 
3  :=  3  -  1 

else  if  Table [ i -1 ][ j ]  <=  Table[i -1  ]  [  j-1  ]  and  Table [ i- 1 ] [ j ]  <=  Table[i] [ j-1 ] 

i  :=  i  -  1 

else 

3  :=  3  -  1 

end  if 


We'll  continue  this  till  we  reach  (0,  0).  Each  move  has  its  own  meaning: 


•  A  horizontal  move  represents  deletion.  That  means  our  Test  sequence  accelerated  during  this  interval. 

•  A  vertical  move  represents  insertion.  That  means  out  Test  sequence  decelerated  during  this  interval. 

•  A  diagonal  move  represents  match.  During  this  period  Test  and  Sample  were  same. 

+ - +- . +- . +- . + - +- . + - + - + 

I  |0|1|1|2|2|3|S| 

+ . + - + - +- . +- . + - + - + - + 

|  9  |  0k,  |  inf  |  inf  |  inf  |  inf  |  inf  |  inf  | 

|  1  |  inf  |  N  fV,  1  |  2  |  4  |  8  | 

I  2  |  inf  |  1  |  1  |\  VVl  1  I  4  | 

I  3  |  inf  |  3  |  3  |  1  |  1  IXI  2  I 

I  S  |  inf  |  7  |  7  |  4  |  4  |  2  |  ' 

1  -  -+ 


+ - + - + - + - + - + - + - + — 

|  5  |  inf  |  11  |  11  |  7  |  7  |  4  |  0 

+ - + - +- . + - + - + - + - + - 

|  S  |  inf  |  15  |  IS  |  10  |  10  |  6  |  $ 

+ - + - + - + - + - + - + - +  -- 

|  6  |  inf  |  20  |  20  |  14  |  14  |  9  | 

+ - + - + - + - + - + - + - + - + 


-  -+ 


-  -  + 


Our  pseudo-code  will  be: 


Procedure  DTW(Sample,  Test) : 
n  :=  Sample . length 
m  :=  Test. length 
Create  Table[n  +  1  ] [m  +  1] 
for  i  f  rom  1  to  n 

Table[i][0]  :=  infinity 
end  for 

for  i  f  rom  1  to  m 

Table[0][i]  :=  infinity 

end  for 

Table [ 0 ] [ 0 ]  :=  0 
for  i  f  rom  1  to  n 

for  j  f  rom  1  to  m 

Table[i][j]  :=  d(Sample[i],  Test[j]) 

+  minimum(Table[ i  1][j-1], 
Table) i] [ j-1 ] , 
Table) i  1  ]  [  j  ]) 

end  for 
end  for 

Return  Table[n  +  1 ] [m  +  1] 


//match 

//insertion 

//deletion 


We  can  also  add  a  locality  constraint.  That  is,  we  require  that  if  Sample)  i]  is  matched  with  Test  [j  ],  then  |i  -  j  |  is 
no  larger  than  w,  a  window  parameter. 
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Complexity: 


The  complexity  of  computing  DTW  is  0(m  *  n)  where  m  and  n  represent  the  length  of  each  sequence.  Faster 
techniques  for  computing  DTW  include  PrunedDTW,  SparseDTW  and  FastDTW. 

Applications: 

•  Spoken  word  recognition 

•  Correlation  Power  Analysis 
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Chapter  51:  Pascal's  Triangle 

Section  51.1:  Pascal  triangle  in  C 

int  i,  space,  rows,  k-0,  count  =  0,  countl  =  0; 
row=5 ; 

for (i=1  i<=rows  ++i) 

{ 

for(space  =  1  ;  space  <=  rows- i  ++space) 

{ 

printf  ( "  "); 

++count  ; 


while(k  ! =  2*i-1 ) 

{ 

if  (count  <=  rows-1 ) 

{ 

printf("%d  ",  i+k); 

++count  ; 

} 

else 

{ 

++count1 ; 

printf ("%d  ",  (i+k  2*count1)); 

} 

++k 

} 

countl  =  count  =  k  =  0 ; 
printf ( " \n" ) ; 


Output 

1 

2  3  2 
3  4  5  4  3 
4  5  6  7  6  5  4 
567898765 
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Chapter  52:  Fast  Fourier  Transform 

The  Real  and  Complex  form  of  DFT  (Discrete  Fourier  Transforms)  can  be  used  to  perform  frequency  analysis  or 
synthesis  for  any  discrete  and  periodic  signals.  The  FFT  (Fast  Fourier  Transform)  is  an  implementation  of  the  DFT 
which  may  be  performed  quickly  on  modern  CPUs. 

Section  52.1:  Radix  2  FFT 


The  simplest  and  perhaps  best-known  method  for  computing  the  FFT  is  the  Radix-2  Decimation  in  Time  algorithm. 
The  Radix-2  FFT  works  by  decomposing  an  N  point  time  domain  signal  into  N  time  domain  signals  each  composed 
of  a  single  point 


1  signal  of 
16  points 


2  signals  of 
8  points 


4  signals  of 
4  points 


8  signals  of 
2  points 


16  signals  of 
1  point 


0  1 

2 

3 

4  5 

6 

7 

8 

9 

10 

11 

CO 

CM 

14  15 

A 

0  2 

4 

6 

8  10 

12 

14 

1 

3 

5 

7 

9  11 

13  15 

A 

A 

0  4 

8 

12 

2  6 

10 

14 

1 

5 

9 

13 

3  7 

11  15 

A 

A 

A 

A 

0  8 

4 

12 

2  10 

6 

14 

1 

9 

5 

13 

3  11 

7  15 

A 

A 

A 

A 

A 

A 

A 

A 

0  8 

4 

12 

2  10 

6 

14 

1 

9 

5 

13 

3  11 

7  15 

Signal  decomposition,  or  ‘decimation  in  time’  is  achieved  by  bit  reversing  the  indices  for  the  array  of  time  domain 
data.  Thus,  for  a  sixteen-point  signal,  sample  1  (Binary  0001 )  is  swapped  with  sample  8  (1 000),  sample  2  (001 0)  is 
swapped  with  4  (01 00)  and  so  on.  Sample  swapping  using  the  bit  reverse  technique  can  be  achieved  simply  in 
software,  but  limits  the  use  of  the  Radix  2  FFT  to  signals  of  length  N  =  2AM. 

The  value  of  a  1  -point  signal  in  the  time  domain  is  equal  to  its  value  in  the  frequency  domain,  thus  this  array  of 
decomposed  single  time-domain  points  requires  no  transformation  to  become  an  array  of  frequency  domain 
points.  The  N  single  points;  however,  need  to  be  reconstructed  into  one  N-point  frequency  spectra.  Optimal 
reconstruction  of  the  complete  frequency  spectrum  is  performed  using  butterfly  calculations.  Each  reconstruction 
stage  in  the  Radix-2  FFT  performs  a  number  of  two  point  butterflies,  using  a  similar  set  of  exponential  weighting 
functions,  WnAR. 
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Va I ues  f rom  Multiply  by 

previous  stage  Weighting  Function 


New 

Values 


The  FFT  removes  redundant  calculations  in  the  Discrete  Fourier  Transform  by  exploiting  the  periodicity  of  WnAR. 
Spectral  reconstruction  is  completed  in  log2(N)  stages  of  butterfly  calculations  giving  X[K];  the  real  and  imaginary 
frequency  domain  data  in  rectangular  form.  To  convert  to  magnitude  and  phase  (polar  coordinates)  requires 
finding  the  absolute  value,  V(Re2  +  Im2),  and  argument,  tan-1  (Im/Re). 

Exponential  Weighting  Factor:  Wjf  =  ed(2*R/N)  =  cos(2kR  /N)  j  sin(2icR  /N) 


N:  Number  of  points  in  the  FFT 

R:  Current  WN  Factor:  depends  on  N,  current  FFT  stage  and 

separation  of  butterflies  in  that  stage 

The  complete  butterfly  flow  diagram  for  an  eight  point  Radix  2  FFT  is  shown  below.  Note  the  input  signals  have 
previously  been  reordered  according  to  the  decimation  in  time  procedure  outlined  previously. 
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X[0]  =  O+jO 
X[1]  =0-j4 
X[2]  =  O+jO 
X[3]  =  O+jO 
X[4]  =  O+jO 
X[5]  =  O+jO 
X[6]  =  O+jO 
X[7]  =  0+j4 


The  FFT  typically  operates  on  complex  inputs  and  produces  a  complex  output.  For  real  signals,  the  imaginary  part 
may  be  set  to  zero  and  real  part  set  to  the  input  signal,  x[n],  however  many  optimisations  are  possible  involving  the 
transformation  of  real-only  data.  Values  of  WnAR  used  throughout  the  reconstruction  can  be  determined  using  the 
exponential  weighting  equation. 

The  value  of  R  (the  exponential  weighting  power)  is  determined  the  current  stage  in  the  spectral  reconstruction  and 
the  current  calculation  within  a  particular  butterfly. 

Code  Example  (C/C++) 

A  C/C++  code  sample  for  computing  the  Radix  2  FFT  can  be  found  below.  This  is  a  simple  implementation  which 
works  for  any  size  N  where  N  is  a  power  of  2.  It  is  approx  3x  slower  than  the  fastest  FFTw  implementation,  but  still  a 
very  good  basis  for  future  optimisation  or  for  learning  about  how  this  algorithm  works. 


#include  <math.h> 


#def ine  PI  3.1415926535897932384626433832795 

#def ine  TWOPI  6.283185307179586476925286766559 

#def ine  Deg2Rad  0.017453292519943295769236907684886 
#def ine  Rad2Deg  57.295779513082320876798154814105 
#def ine  log10_2  0.30102999566398119521373889472449 

#def ine  log10_2_INV  3.3219280948873623478703194294948 


//  PI  for  sine/cos  calculations 
//  2*PI  for  sine/cos  calculations 
//  Degrees  to  Radians  factor 
//  Radians  to  Degrees  factor 
//  Logl 0  of  2 
//  1 /Logl 0(2) 


//  complex  variable  structure  (double  precision) 
struct  complex 
{ 

public : 

double  Re,  Im;  //  Not  so  complicated  after  all 

}  ; 


//  Returns  true  if  N  is  a  power  of  2 
bool  isPwrTwo(int  N,  int  *M) 

{ 

*M  =  (int)ceil(log10( (double)N)  *  logl 0_2_INV) ; //  M  is  number  of  stages  to  perform.  2AM  =  N 
int  NN  =  (int)pow(2.0,  *M)  ; 
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if  ((NN  !=  N)  ||  (NN  ==  0))  //  Check  N  is  a  power  of  2. 
return  false; 

return  true; 

} 

void  rad2FFT(int  N,  complex  *x,  complex  *DFT) 

{ 

int  M  =  0 ; 


//  Check  if  power  of  two.  If  not,  exit 
if  ( ! isPwrTwo(N,  &M)) 

throw  "Rad2FFT():  N  must  be  a  power  of  2  for  Radix  FFT"  ; 


//  Integer  Variables 


int  BSep 
int  BWidth 
int  P 
int  j ; 

int  stage  =  1  ; 

( 1  to  M) . 

int  Hilndex; 
butterfly  calc 

unsigned  int  iaddr; 
int  ii 

int  MM1  =  M  -  1  ; 


//  BSep  is  memory  spacing  between  butterflies 

//  BWidth  is  memory  spacing  of  opposite  ends  of  the  butterfly 

//  P  is  number  of  similar  Wn ' s  to  be  used  in  that  stage 

//  j  is  used  in  a  loop  to  perform  all  calculations  in  each  stage 

//  stage  is  the  stage  number  of  the  FFT.  There  are  M  stages  in  total 

//  Hilndex  is  the  index  of  the  DFT  array  for  the  top  value  of  each 

//  bitmask  for  bit  reversal 

//  Integer  bitfield  for  bit  reversal  (Decimation  in  Time) 


unsigned  int  i 
int  1; 

unsigned  int  nMax  =  (unsigned  int)N; 


//  Double  Precision  Variables 

double  TwoPi_N  =  TWOPI  /  (double)N;  //  constant  to  save  computational  time.  =  2*PI  /  N 
double  TwoPi_NP; 


//  complex  Variables 

complex  WN 
complex  TEMP 
complex  *pDFT  =  DFT 
complex  *pLo 
complex  *pHi 
complex  *pX 


(See  'struct  complex') 

//  Wn  is  the  exponential  weighting  function  in  the  form  a  +  jb 
//  TEMP  is  used  to  save  computation  in  the  butterfly  calc 
//  Pointer  to  first  elements  in  DFT  array 
//  Pointer  for  lo  /  hi  value  of  butterfly  calcs 

//  Pointer  to  x[n] 


//  Decimation  In  Time 
for  (i  =  0;  i  <  nMax; 
{ 

pX  =  x  +  i 
ii  =  0; 
iaddr  =  i 
for  (1  =  0;  1  <  M 
{ 


-  x[n]  sample  sorting 
i++,  DFT++) 

//  Calculate  current  x[n]  from  base  address  *x  and  index  i. 
//  Reset  new  address  for  DFT[n] 

//  Copy  i  for  manipulations 
1++)  //  Bit  reverse  i  and  store  in  ii... 


if  (iaddr  &  0x01)  //  Detemine  least  significant  bit 

ii  +=  (1  <<  (MM1  -  1));  //  Increment  ii  by  2A(M-1-1)  if  lsb  was  1 

iaddr  >>=  1;  //  right  shift  iaddr  to  test  next  bit.  Use  logical 

operations  for  speed  increase 
if  (! iaddr) 
break  ; 


} 

DFT  =  pDFT  +  ii  //  Calculate  current  DFT[n]  from  base  address  *pDFT  and  bit 

reversed  index  ii 
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DFT->Re  =  pX->Re 


//  Update  the  complex  array  with  address  sorted  time  domain  signal 


x[n] 

DFT->Im  =  pX->Im;  //  NB :  Imaginary  is  always  zero 

} 

//  FFT  Computation  by  butterfly  calculation 

for  (stage  =  1;  stage  <=  M  stage++)  //  Loop  for  M  stages,  where  2AM  =  N 

{ 

BSep  =  (int)(pow(2,  stage));  //  Separation  between  butterflies  =  2Astage 
P  =  N  /  BSep  //  Similar  Wn 1 s  in  this  stage  =  N/Bsep 

BWidth  =  BSep  /  2;  //  Butterfly  width  (spacing  between  opposite  points)  =  Separation  / 

2. 


TwoPi_NP  =  TwoPi_N*P 


for  (j  =0;  j  < 
{ 

if  (j  ! =  0) 
{ 

//WN.Re 
WN . Re  = 
WN.Im  = 

} 


BWidth  ]++)  //  Loop  for  j  calculations  per  butterfly 

//  Save  on  calculation  if  R  =  0,  as  WNA0  =  (1  +  j0) 
=  cos(TwoPi_NP*j ) 

cos(TwoPi_N*P*j ) ;  //  Calculate  Wn  (Real  and  Imaginary) 

-sin(TwoPi_N*P*j ) ; 


for  (Hilndex  =  j;  Hilndex  <  N 
butterflies  per  stage 
{ 

pHi  =  pDFT  +  Hilndex; 
pLo  =  pHi  +  BWidth 

for  spacing  between  elements) 


Hilndex  +=  BSep)  //  Loop  for  Hilndex  Step  BSep 


//  Point  to  higher  value 

//  Point  to  lower  value  (Note  VC++  adjusts 


with  Wn 


} 


} 


if  (j  !=  0)  //  If  exponential  power  is  not  zero... 

{ 

//CMult(pLo,  &WN,  &TEMP) ;  //  Perform  complex  multiplication  of  Lovalue 

TEMP. Re  =  (pLo->Re  *  WN.Re)  -  (pLo->Im  *  WN.Im); 

TEMP.Im  =  (pLo->Re  *  WN.Im)  +  (pLo->Im  *  WN.Re); 


//CSub  (pHi,  &TEMP,  pLo); 

pLo->Re  =  pHi->Re  -  TEMP. Re;  //  Find  new  Lovalue  (complex  subtraction) 

pLo->Im  =  pHi->Im  -  TEMP.Im; 


//CAdd  (pHi,  &TEMP,  pHi); 

pHi->Re  =  (pHi->Re  +  TEMP. Re); 
pHi->Im  =  (pHi->Im  +  TEMP.Im); 

} 

else 


TEMP. Re  =  pLo->Re 
TEMP.Im  =  pLo->Im; 


//  Find  new  Hivalue  (complex  addition) 


//CSub  (pHi,  &TEMP,  pLo) ; 

pLo->Re  =  pHi->Re  -  TEMP. Re; 
pLo->Im  =  pHi->Im  -  TEMP.Im; 


//  Find  new  Lovalue  (complex  subtraction) 


//CAdd  (pHi,  &TEMP,  pHi);  //  Find  new  Hivalue  (complex  addition) 

pHi->Re  =  (pHi->Re  +  TEMP. Re); 
pHi->Im  =  (pHi->Im  +  TEMP.Im); 
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//  Null  all  pointers 


pLo  =  0; 
pHi  =  0 ; 
pDFT  =  0; 

DFT  =  0; 
pX  =  0; 

} 

Section  52.2:  Radix  2  Inverse  FFT 

Due  to  the  strong  duality  of  the  Fourier  Transform,  adjusting  the  output  of  a  forward  transform  can  produce  the 
inverse  FFT.  Data  in  the  frequency  domain  can  be  converted  to  the  time  domain  by  the  following  method: 

1 .  Find  the  complex  conjugate  of  the  frequency  domain  data  by  inverting  the  imaginary  component  for  all 
instances  of  K. 

2.  Perform  the  forward  FFT  on  the  conjugated  frequency  domain  data. 

3.  Divide  each  output  of  the  result  of  this  FFT  by  N  to  give  the  true  time  domain  value. 

4.  Find  the  complex  conjugate  of  the  output  by  inverting  the  imaginary  component  of  the  time  domain  data  for 
all  instances  of  n. 

Note:  both  frequency  and  time  domain  data  are  complex  variables.  Typically  the  imaginary  component  of  the  time 
domain  signal  following  an  inverse  FFT  is  either  zero,  or  ignored  as  rounding  error.  Increasing  the  precision  of  variables 
from  32-bit  float  to  64-bit  double,  or  128-bit  long  double  significantly  reduces  rounding  errors  produced  by  several 
consecutive  FFT  operations. 

Code  Example  (C/C++) 


#include  <math.h> 


#def ine  PI  3.1415926535897932384626433832795 

#def ine  TWOPI  6.283185307179586476925286766559 

#def ine  Deg2Rad  0.017453292519943295769236907684886 
#def ine  Rad2Deg  57.295779513082320876798154814105 
#def ine  log10_2  0.30102999566398119521373889472449 

#def ine  log10_2_INV  3.3219280948873623478703194294948 


//  PI  for  sine/cos  calculations 
//  2*PI  for  sine/cos  calculations 
//  Degrees  to  Radians  factor 
//  Radians  to  Degrees  factor 
//  Logl 0  of  2 
//  1 /Logl 0(2) 


//  complex  variable  structure  (double  precision) 
struct  complex 
{ 

public : 

double  Re,  Im;  //  Not  so  complicated  after  all 

}  ; 


void  rad2InverseFFT(int  N,  complex  *x,  complex  *DFT) 

{ 

//  M  is  number  of  stages  to  perform.  2AM  =  N 
double  Mx  =  ( logl  0(  (double) N)  /  logl 0( (double)2) ) ; 
int  a  =  (int) (ceil(pow(2 . 0,  Mx))); 
int  status  =  0; 

if  (a  !=  N)  //  Check  N  is  a  power  of  2 

{ 

x  =  0  ; 

DFT  =  0; 

throw  " rad2InverseFFT( ) :  N  must  be  a  power  of  2  for  Radix  2  Inverse  FFT"; 

} 


complex  *pDFT  =  DFT 
complex  *pX  =  x; 
double  NN  =  1  /  (double)N 


//  Reset  vector  for  DFT  pointers 
//  Reset  vector  for  x[n]  pointer 
//  Scaling  factor  for  the  inverse  FFT 
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for  (int  i  =  0  i  <  N  i++, 
DFT->Im  *=  -1 ; 


DFT++) 

//  Find  the  complex  conjugate  of  the  Frequency  Spectrum 


DFT  =  pDFT  //  Reset  Freq  Domain  Pointer 

rad2FFT(N,  DFT,  x) ;  //  Calculate  the  forward  FFT  with  variables  switched  (time  &  freq) 


} 


int  i 

complex*  x; 

for  (  i  =  0,  x  =  pX;  i 
x->Re  *=  NN;  // 

x->Im  *=  -1 ;  // 

} 


<  N  i++,  x++){ 
Divide  time  domain 
Change  the  sign  of 


by  N  for  correct  amplitude  scaling 
ImX 
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Chapter  53:  Algo:-  Print  a  m*n  matrix  in 
square  wise 

Check  sample  input  and  output  below. 

Section  53.1:  Sample  Example 

Input : 

14  15  16  17  18  21 
19  10  20  11  54  36 
64  55  44  23  80  39 
91  92  93  94  95  42 

Output: 

print  value  in  index 

14  15  16  17  18  21  36  39  42  95  94  93  92  91  64  19  10  20  11  54  80  23  44  55 
or  print  index 

00  01  02  03  04  05  15  25  35  34  33  32  31  30  20  10  11  12  13  14  24  23  22  21 


Section  53.2:  Write  the  generic  code 


function  noOf Looping(m,  n)  { 
if(m  >  n)  { 

smallestValue  =  n 

}  else  { 

smallestValue  =  m; 

} 

if (smallestValue  X  2  ==  0)  { 

return  smallestValue/2  ; 

}  else  { 

return  (smallestValue+1 ) /2 ; 

} 


function  squarePrint(m, n)  { 

var  looping  =  noOf Looping (m, n)  ; 
for  (var  i  =  0;  i  <  looping  i++)  { 

for  (  var  j=i  j  <  m  -  1  -  i  j++) 
console. log(i+' '+]) ; 

} 

for(var  k  =  i  k  <  n  1  -  i  k++) 
console. log(k+' '+]) ; 

} 

for  (  var  1  =  j ;  1  >  i  l-){ 
console . log (k+ ' '+1) ; 

} 

for  (  var  x  =  k  x  >  i  x--)  { 
console. log(x+' '+1) ; 

} 

} 

} 


{ 

{ 


squarePrint(6,4) ; 
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Chapter  54:  Check  two  strings  are 
anagrams 

Two  string  with  same  set  of  character  is  called  anagram.  I  have  used  javascript  here. 

We  will  create  an  hash  of  strl  and  increase  count  +1 .  We  will  loop  on  2nd  string  and  check  all  characters  are  there 

in  hash  and  decrease  value  of  hash  key.  Check  all  value  of  hash  key  are  zero  will  be  anagram. 

Section  54.1:  Sample  input  and  output 

Exl: 

let  strl  =  ' stackoverflow' ; 

let  str2  =  ' f lowerovstack ' ; 

These  strings  are  anagrams. 

//  Create  Hash  from  strl  and  increase  one  count. 

hashMap  =  { 
s  :  1 , 
t  :  1 , 
a  :  1 , 
c  :  1 , 
k  :  1 , 
o  :  2, 
v  :  1 , 
e  :  1 , 
r  :  1 , 

f  1, 

1  1 , 

w  :  1 

} 

You  can  see  hashKey  'o'  is  containing  value  2  because  o  is  2  times  in  string. 

Now  loop  over  str2  and  check  for  each  character  are  present  in  hashMap,  if  yes,  decrease  value  of  hashMap  Key, 

else  return  false  (which  indicate  it's  not  anagram). 

hashMap  =  { 
s  :  0, 
t  :  0, 
a  :  0, 
c  :  0, 
k  0, 
o  :  0, 
v  :  0, 
e  :  0, 
r  :  0, 

f  0, 

1  :  0, 

w  :  0 

} 

Now,  loop  over  hashMap  object  and  check  all  values  are  zero  in  the  key  of  hashMap. 
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In  our  case  all  values  are  zero  so  its  a  anagram. 


Section  54.2:  Generic  Code  for  Anagrams 


(function( ) { 

var  hashMap  =  {}; 

function  isAnagram  (strl,  str2)  { 

if (strl . length  !==  str2 . length) { 

return  false 

} 

//  Create  hash  map  of  strl  character  and  increase  value  one  (+1 ) . 
createStr1HashMap(str1 )  ; 

//  Check  str2  character  are  key  in  hash  map  and  decrease  value  by  one(-l); 
var  valueExist  =  createStr2HashMap(str2)  ; 

//  Check  all  value  of  hashMap  keys  are  zero,  so  it  will  be  anagram. 
return  isStringsAnagram(valueExist) ; 

> 

function  createStrl HashMap  (strl)  { 

[ ] .map.call(str1 ,  function(value,  index,  array){ 

hashMap [value  =  value  in  hashMap  ?  (hashMap [value  +1)  :  1; 
return  value 

}); 

} 


function  createStr2HashMap  (str2)  { 

var  valueExist  =  [] .every .call(str2,  function(value,  index,  array){ 
if(value  in  hashMap)  { 

hashMap [value  =  hashMap[ value  -  1  ; 

} 

return  value  in  hashMap 

}); 

return  valueExist ; 

} 


function  isStringsAnagram  (valueExist)  { 
if ( ! valueExist)  { 

return  valueExist; 

}  else  { 

var  isAnagram; 
for  (var  i  in  hashMap)  { 
if (hashMap[ i]  ! ==  0)  { 
isAnagram  =  false 
break 
}  else  { 

isAnagram  =  true 

} 

} 


return  isAnagram; 

} 

} 

isAnagram( ' stackoverf low' ,  ' f lowerovstack ' ) ;  //  true 
isAnagram( ' stackoverf low' ,  ' f lowervvstack ' ) ;  //  false 
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})(); 


Time  complexity:  3n  i.e  0(n). 
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Chapter  55:  Applications  of  Dynamic 
Programming 

The  basic  idea  behind  dynamic  programming  is  breaking  a  complex  problem  down  to  several  small  and  simple 
problems  that  are  repeated.  If  you  can  identify  a  simple  subproblem  that  is  repeatedly  calculated,  odds  are  there  is 
a  dynamic  programming  approach  to  the  problem. 

As  this  topic  is  titled  Applications  of  Dynamic  Programming,  it  will  focus  more  on  applications  rather  than  the  process 
of  creating  dynamic  programming  algorithms. 

Section  55.1:  Fibonacci  Numbers 

Fibonacci  Numbers  are  a  prime  subject  for  dynamic  programming  as  the  traditional  recursive  approach  makes  a  lot 
of  repeated  calculations.  In  these  examples  I  will  be  using  the  base  case  of  f  ( 0 )  =  f(i)  =  i. 

Here  is  an  example  recursive  tree  for  fibonacci(4),  note  the  repeated  computations: 


/(2)  /(i)  /(i)  m 


/( i)  /(0) 

Non-Dynamic  Programming  0(2An)  Runtime  Complexity,  0(n)  Stack  complexity 

def  fibonacci(n) : 
if  n  <  2: 

return  1 

return  fibonacci(n-1 )  +  f ibonacci(n-2) 

This  is  the  most  intuitive  way  to  write  the  problem.  At  most  the  stack  space  will  be  0(n)  as  you  descend  the  first 
recursive  branch  making  calls  to  fibonacci(n-1 )  until  you  hit  the  base  case  n  <  2. 

The  0(2An)  runtime  complexity  proof  that  can  be  seen  here:  Computational  complexity  of  Fibonacci  Sequence.  The 
main  point  to  note  is  that  the  runtime  is  exponential,  which  means  the  runtime  for  this  will  double  for  every 
subsequent  term,  f  ibonacci(1 5)  will  take  twice  as  long  as  f  ibonacci(14). 

Memoized  0(n)  Runtime  Complexity,  0(n)  Space  complexity,  0(n)  Stack  complexity 

memo  =  [ ] 

memo.append(1 )  #  f ( 1 )  =  1 
memo.append(1 )  #  f ( 2 )  =  1 

def  fibonacci(n)  : 

if  len(memo)  >  n 
return  memojn 
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result  =  fibonacci(n -1 )  +  fibonacci(n  2) 
memo.append(result)  #  f(n)  =  f(n-1)  +  f(n  2) 

return  result 

With  the  memoized  approach  we  introduce  an  array  that  can  be  thought  of  as  all  the  previous  function  calls.  The 
location  memo[n]  is  the  result  of  the  function  call  fibonacci(n).  This  allows  us  to  trade  space  complexity  of  0(n)  for 
a  0(n)  runtime  as  we  no  longer  need  to  compute  duplicate  function  calls. 

Iterative  Dynamic  Programming  0(n)  Runtime  complexity,  0(n)  Space  complexity,  No  recursive  stack 

def  fibonacci(n) : 

memo  =  [1,1]  #  f(0)  =  1,  f(1)  =  1 

for  i  in  range(2,  n+1 ) : 

memo.append(memo[i -1  ]  +  memo[i-2]) 

return  memo[n] 

If  we  break  the  problem  down  into  it's  core  elements  you  will  notice  that  in  order  to  compute  fibonacci(n)  we 
need  f  ibonacci(n  1 )  and  f  ibonacci(n-2).  Also  we  can  notice  that  our  base  case  will  appear  at  the  end  of  that 
recursive  tree  as  seen  above. 

With  this  information,  it  now  makes  sense  to  compute  the  solution  backwards,  starting  at  the  base  cases  and 
working  upwards.  Now  in  order  to  calculate  fibonacci(n)  we  first  calculate  all  the  fibonacci  numbers  up  to  and 
through  n. 

This  main  benefit  here  is  that  we  now  have  eliminated  the  recursive  stack  while  keeping  the  0(n)  runtime. 
Unfortunately,  we  still  have  an  0(n)  space  complexity  but  that  can  be  changed  as  well. 

Advanced  Iterative  Dynamic  Programming  0(n)  Runtime  complexity,  0(1 )  Space  complexity,  No  recursive  stack 

def  fibonacci (n) : 

memo  =  [1,1]  #  f(1)  =  1,  f(2)  =  1 

for  i  in  range  (2,  n) : 

memo[i%2]  =  memo[0]  +  memo[1] 

return  memo[n%2] 

As  noted  above,  the  iterative  dynamic  programming  approach  starts  from  the  base  cases  and  works  to  the  end 
result.  The  key  observation  to  make  in  order  to  get  to  the  space  complexity  to  0(1 )  (constant)  is  the  same 
observation  we  made  for  the  recursive  stack  -  we  only  need  fibonacci(n  1 )  and  fibonacci(n-2)  to  build 
fibonacci(n).  This  means  that  we  only  need  to  save  the  results  for  fibonacci(n-1 )  and  fibonacci(n-2)  at  any 
point  in  our  iteration. 

To  store  these  last  2  results  I  use  an  array  of  size  2  and  simply  flip  which  index  I  am  assigning  to  by  using  i  %  2 
which  will  alternate  like  so:  0,  1,  0,  1,  0,  1,  ...,  i  %  2. 

I  add  both  indexes  of  the  array  together  because  we  know  that  addition  is  commutative  (5  +  6  =  lland6  +  5  == 

1 1 ).  The  result  is  then  assigned  to  the  older  of  the  two  spots  (denoted  by  i  %  2).  The  final  result  is  then  stored  at 
the  position  n%2 


Notes 


•  It  is  important  to  note  that  sometimes  it  may  be  best  to  come  up  with  a  iterative  memoized  solution  for 
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functions  that  perform  large  calculations  repeatedly  as  you  will  build  up  a  cache  of  the  answer  to  the 
function  calls  and  subsequent  calls  may  be  0(  1 )  if  it  has  already  been  computed. 
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Appendix  A:  Pseudocode 

Section  A.1:  Variable  affectations 

You  could  describe  variable  affectation  in  different  ways. 

Typed 

int  a  =  1 

int  a  :=  1 

let  int  a  =  1 

int  a  <-  1 


No  type 

a  =  1 
a  :=  1 

let  a  =  1 
a  <-  1 

Section  A.2:  Functions 

As  long  as  the  function  name,  return  statement  and  parameters  are  clear,  you're  fine. 

def  incr  n 

return  n  +  1 


or 

let  incr(n)  =  n  +  1 

or 

function  incr  (n) 

return  n  +  1 

are  all  quite  clear,  so  you  may  use  them.  Try  not  to  be  ambiguous  with  a  variable  affectation 
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